摘要
近年来,随机方差缩减类方法在解决大规模机器学习问题中取得很大成功,自适应步长技术的引入减轻了该类方法的调参负担。针对自适应步长的方差缩减算法SVRG-BB,指出其算法设计带来了“进展-自适应步长有效性”的权衡问题。因此引入Katyusha动量以更好地处理该权衡问题,并且在强凸假设下证明由此得到的SVRG-BB-Katyusha算法的线性收敛性质。之后基于“贪婪”思想,提出稀疏地使用Katyusha动量的SVRG-BB-Katyusha-SPARSE算法。在公开数据集上的数值实验结果表明,提出的2个改进算法较SVRG-BB有较稳定的优势,即在达到一定外循环数时优化间隙有若干个数量级的减小。
Stochastic variance reduction methods have been successful in solving large scale machine learning problems,and researchers cooperate them with adaptive stepsize schemes to further alleviate the burden of parameter-tuning.In this article,we propose that there exists a trade-off between progress and effectiveness of adaptive stepsize arising in the SVRG-BB algorithm.To enhance the practical performance of SVRG-BB,we introduce the Katyusha momentum to handle the aforementioned trade-off.The linear convergence rate of the resulting SVRG-BB-Katyusha algorithm is proven under strong convexity condition.Moreover,we propose SVRG-BB-Katyusha-SPARSE algorithm which uses Katyusha momentum sparsely in the inner iterations.Numerical experiments are given to illustrate that the proposed algorithms have promising advantages over SVRG-BB,in the sense that the optimality gaps of the proposed algorithms are smaller than the optimality gap of SVRG-BB by orders of magnitude.
作者
刘海
郭田德
韩丛英
LIU Hai;GUO Tiande;HAN Congying(School of Mathematical Sciences,University of Chinese Academy of Sciences,Beijing 100049,China)
出处
《中国科学院大学学报（中英文）》
CAS
CSCD
北大核心
2024年第5期577-588,共12页
Journal of University of Chinese Academy of Sciences
基金
国家重点研发计划(2021YFA1000403)
国家自然科学基金(11991022,U23B2012)
中央高校基本科研业务费专项(E1E40104X2)资助。
