This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting.A class of stochastic momentum methods,including stochastic gradient descent,heavy ball and Neste...This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting.A class of stochastic momentum methods,including stochastic gradient descent,heavy ball and Nesterov’s accelerated gradient,is analyzed in a general framework under mild assumptions.Based on the convergence result of expected gradients,the authors prove the almost sure convergence by a detailed discussion of the effects of momentum and the number of upcrossings.It is worth noting that there are not additional restrictions imposed on the objective function and stepsize.Another improvement over previous results is that the existing Lipschitz condition of the gradient is relaxed into the condition of H?lder continuity.As a byproduct,the authors apply a localization procedure to extend the results to stochastic stepsizes.展开更多
We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments w...We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number,and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold.Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large,ill-conditioned problems.展开更多
基金supported by the National Natural Science Foundation of China (Nos. 11631004,12031009)the National Key R&D Program of China (No. 2018YFA0703900)。
文摘This paper is concerned with convergence of stochastic gradient algorithms with momentum terms in the nonconvex setting.A class of stochastic momentum methods,including stochastic gradient descent,heavy ball and Nesterov’s accelerated gradient,is analyzed in a general framework under mild assumptions.Based on the convergence result of expected gradients,the authors prove the almost sure convergence by a detailed discussion of the effects of momentum and the number of upcrossings.It is worth noting that there are not additional restrictions imposed on the objective function and stepsize.Another improvement over previous results is that the existing Lipschitz condition of the gradient is relaxed into the condition of H?lder continuity.As a byproduct,the authors apply a localization procedure to extend the results to stochastic stepsizes.
文摘We develop a generalization of Nesterov’s accelerated gradient descent method which is designed to deal with orthogonality constraints.To demonstrate the effectiveness of our method,we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number,and also compare with existing state-of-the-art quasi-Newton methods on the Stiefel manifold.Our experiments show that our method outperforms existing state-of-the-art quasi-Newton methods on some large,ill-conditioned problems.
