Sentiment analysis plays an important role in distilling and clarifying content from movie reviews,aiding the audience in understanding universal views towards the movie.However,the abundance of reviews and the risk o...Sentiment analysis plays an important role in distilling and clarifying content from movie reviews,aiding the audience in understanding universal views towards the movie.However,the abundance of reviews and the risk of encountering spoilers pose challenges for efcient sentiment analysis,particularly in Arabic content.Tis study proposed a Stochastic Gradient Descent(SGD)machine learning(ML)model tailored for sentiment analysis in Arabic and English movie reviews.SGD allows for fexible model complexity adjustments,which can adapt well to the Involvement of Arabic language data.Tis adaptability ensures that the model can capture the nuances and specifc local patterns of Arabic text,leading to better performance.Two distinct language datasets were utilized,and extensive pre-processing steps were employed to optimize the datasets for analysis.Te proposed SGD model,designed to accommodate the nuances of each language,aims to surpass existing models in terms of accuracy and efciency.Te SGD model achieves an accuracy of 84.89 on the Arabic dataset and 87.44 on the English dataset,making it the top-performing model in terms of accuracy on both datasets.Tis indicates that the SGD model consistently demonstrates high accuracy levels across Arabic and English datasets.Tis study helps deepen the understanding of sentiments across various linguistic datasets.Unlike many studies that focus solely on movie reviews,the Arabic dataset utilized here includes hotel reviews,ofering a broader perspective.展开更多
Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for ga...Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.展开更多
With the increasing prevalence of high-order systems in engineering applications, these systems often exhibitsignificant disturbances and can be challenging to model accurately. As a result, the active disturbance rej...With the increasing prevalence of high-order systems in engineering applications, these systems often exhibitsignificant disturbances and can be challenging to model accurately. As a result, the active disturbance rejectioncontroller (ADRC) has been widely applied in various fields. However, in controlling plant protection unmannedaerial vehicles (UAVs), which are typically large and subject to significant disturbances, load disturbances andthe possibility of multiple actuator faults during pesticide spraying pose significant challenges. To address theseissues, this paper proposes a novel fault-tolerant control method that combines a radial basis function neuralnetwork (RBFNN) with a second-order ADRC and leverages a fractional gradient descent (FGD) algorithm.We integrate the plant protection UAV model’s uncertain parameters, load disturbance parameters, and actuatorfault parameters and utilize the RBFNN for system parameter identification. The resulting ADRC exhibits loaddisturbance suppression and fault tolerance capabilities, and our proposed active fault-tolerant control law hasLyapunov stability implications. Experimental results obtained using a multi-rotor fault-tolerant test platformdemonstrate that the proposed method outperforms other control strategies regarding load disturbance suppressionand fault-tolerant performance.展开更多
We prove,under mild conditions,the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model,both in batch gradient descent and stochastic gradient descent.We also discuss a ...We prove,under mild conditions,the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model,both in batch gradient descent and stochastic gradient descent.We also discuss a Riemannian version of the Adam algorithm.We show numerical simulations of these algorithms on various benchmarks.展开更多
A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and...A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.展开更多
Matrix completion is the extension of compressed sensing.In compressed sensing,we solve the underdetermined equations using sparsity prior of the unknown signals.However,in matrix completion,we solve the underdetermin...Matrix completion is the extension of compressed sensing.In compressed sensing,we solve the underdetermined equations using sparsity prior of the unknown signals.However,in matrix completion,we solve the underdetermined equations based on sparsity prior in singular values set of the unknown matrix,which also calls low-rank prior of the unknown matrix.This paper firstly introduces basic concept of matrix completion,analyses the matrix suitably used in matrix completion,and shows that such matrix should satisfy two conditions:low rank and incoherence property.Then the paper provides three reconstruction algorithms commonly used in matrix completion:singular value thresholding algorithm,singular value projection,and atomic decomposition for minimum rank approximation,puts forward their shortcoming to know the rank of original matrix.The Projected Gradient Descent based on Soft Thresholding(STPGD),proposed in this paper predicts the rank of unknown matrix using soft thresholding,and iteratives based on projected gradient descent,thus it could estimate the rank of unknown matrix exactly with low computational complexity,this is verified by numerical experiments.We also analyze the convergence and computational complexity of the STPGD algorithm,point out this algorithm is guaranteed to converge,and analyse the number of iterations needed to reach reconstruction error.Compared the computational complexity of the STPGD algorithm to other algorithms,we draw the conclusion that the STPGD algorithm not only reduces the computational complexity,but also improves the precision of the reconstruction solution.展开更多
In a wireless sensor network[1],the operation of a node depends on the battery power it carries.Because of the environmental reasons,the node cannot replace the battery.In order to improve the life cycle of the networ...In a wireless sensor network[1],the operation of a node depends on the battery power it carries.Because of the environmental reasons,the node cannot replace the battery.In order to improve the life cycle of the network,energy becomes one of the key problems in the design of the wireless sensor network(WSN)routing protocol[2].This paper proposes a routing protocol ERGD based on the method of gradient descent that can minimizes the consumption of energy.Within the communication radius of the current node,the distance between the current node and the next hop node is assumed that can generate a projected energy at the distance from the current node to the base station(BS),this projected energy and the remaining energy of the next hop node is the key factor in finding the next hop node.The simulation results show that the proposed protocol effectively extends the life cycle of the network and improves the reliability and fault tolerance of the system.展开更多
The distribution of sampling data influences completeness of rule base so that extrapolating missing rules is very difficult. Based on data mining, a self-learning method is developed for identifying fuzzy model and e...The distribution of sampling data influences completeness of rule base so that extrapolating missing rules is very difficult. Based on data mining, a self-learning method is developed for identifying fuzzy model and extrapolating missing rules, by means of confidence measure and the improved gradient descent method. The proposed approach can not only identify fuzzy model, update its parameters and determine optimal output fuzzy sets simultaneously, but also resolve the uncontrollable problem led by the regions that data do not cover. The simulation results show the effectiveness and accuracy of the proposed approach with the classical truck backer-upper control problem verifying.展开更多
A gradient descent algorithm with adjustable parameter for attitude estimation is developed,aiming at the attitude measurement for small unmanned aerial vehicle(UAV)in real-time flight conditions.The accelerometer and...A gradient descent algorithm with adjustable parameter for attitude estimation is developed,aiming at the attitude measurement for small unmanned aerial vehicle(UAV)in real-time flight conditions.The accelerometer and magnetometer are introduced to construct an error equation with the gyros,thus the drifting characteristics of gyroscope can be compensated by solving the error equation utilized by the gradient descent algorithm.Performance of the presented algorithm is evaluated using a self-proposed micro-electro-mechanical system(MEMS)based attitude heading reference system which is mounted on a tri-axis turntable.The on-ground,turntable and flight experiments indicate that the estimation attitude has a good accuracy.Also,the presented system is compared with an open-source flight control system which runs extended Kalman filter(EKF),and the results show that the attitude control system using the gradient descent method can estimate the attitudes for UAV effectively.展开更多
Motion estimation is an important part of the MPEG- 4 encoder, due to its significant impact on the bit rate and the output quality of the encoder sequence. Unfortunately this feature takes a significant part of the e...Motion estimation is an important part of the MPEG- 4 encoder, due to its significant impact on the bit rate and the output quality of the encoder sequence. Unfortunately this feature takes a significant part of the encoding time especially when the straightforward full search(FS) algorithm is used. In this paper, a new algorithm named diamond block based gradient descent search (DBBGDS) algorithm, which is significantly faster than FS and gives similar quality of the output sequence, is proposed. At the same time, some other algorithms, such as three step search (TSS), improved three step search (ITSS), new three step search (NTSS), four step search (4SS), cellular search (CS) , diamond search (DS) and block based gradient descent search (BBGDS), are adopted and compared with DBBGDS. As the experimental results show, DBBGDS has its own advantages. Although DS has been adopted by the MPEG- 4 VM, its output sequence quality is worse than that of the proposed algorithm while its complexity is similar to the proposed one. Compared with BBGDS, the proposed algorithm can achieve a better output quality.展开更多
Rockburst is a phenomenon in which free surfaces are formed during excavation,which subsequently causes the sudden release of energy in the construction of mines and tunnels.Light rockburst only peels off rock slices ...Rockburst is a phenomenon in which free surfaces are formed during excavation,which subsequently causes the sudden release of energy in the construction of mines and tunnels.Light rockburst only peels off rock slices without ejection,while severe rockburst causes casualties and property loss.The frequency and degree of rockburst damage increases with the excavation depth.Moreover,rockburst is the leading engineering geological hazard in the excavation process,and thus the prediction of its intensity grade is of great significance to the development of geotechnical engineering.Therefore,the prediction of rockburst intensity grade is one problem that needs to be solved urgently.By comprehensively considering the occurrence mechanism of rockburst,this paper selects the stress index(σθ/σc),brittleness index(σ_(c)/σ_(t)),and rock elastic energy index(Wet)as the rockburst evaluation indexes through the Spearman coefficient method.This overcomes the low accuracy problem of a single evaluation index prediction method.Following this,the BGD-MSR-DNN rockburst intensity grade prediction model based on batch gradient descent and a multi-scale residual deep neural network is proposed.The batch gradient descent(BGD)module is used to replace the gradient descent algorithm,which effectively improves the efficiency of the network and reduces the model training time.Moreover,the multi-scale residual(MSR)module solves the problem of network degradation when there are too many hidden layers of the deep neural network(DNN),thus improving the model prediction accuracy.The experimental results reveal the BGDMSR-DNN model accuracy to reach 97.1%,outperforming other comparable models.Finally,actual projects such as Qinling Tunnel and Daxiangling Tunnel,reached an accuracy of 100%.The model can be applied in mines and tunnel engineering to realize the accurate and rapid prediction of rockburst intensity grade.展开更多
A new algorithm to exploit the learning rates of gradient descent method is presented, based on the second-order Taylor expansion of the error energy function with respect to learning rate, at some values decided by &...A new algorithm to exploit the learning rates of gradient descent method is presented, based on the second-order Taylor expansion of the error energy function with respect to learning rate, at some values decided by "award-punish" strategy. Detailed deduction of the algorithm applied to RBF networks is given. Simulation studies show that this algorithm can increase the rate of convergence and improve the performance of the gradient descent method.展开更多
Regulating the power output for a power plant as demand for electricity fluctuates throughout the day is important for both economic purpose and the safety of the generator. In this work, gradient descent method toget...Regulating the power output for a power plant as demand for electricity fluctuates throughout the day is important for both economic purpose and the safety of the generator. In this work, gradient descent method together with regularization is investigated to study the electricity output related to vacuum level and temperature in the turbine. Ninety percent of the data was used to train the regression parameters while the remaining ten percent was used for validation. Final results showed that 99% accuracy could be obtained with this method. This opens a new window for electricity output prediction for power plants.展开更多
In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(1/√T) convergence ...In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(1/√T) convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the FashionMinst, CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic step size improves test accuracy by 0.9% for the CIFAR100 dataset when we utilize a convolutional neural network (CNN) model.展开更多
In many applications involving data streams,the sequences of data arise from highly dynamic and often unstable real-life processes,rendering untenable the standard assumption that current and future data come from the...In many applications involving data streams,the sequences of data arise from highly dynamic and often unstable real-life processes,rendering untenable the standard assumption that current and future data come from the same distribution.In response,new methodologies,such as dynamic online learning,have been proposed in order to account for the nonstationary features in the datagenerating process.Motivated by the stability and statistical efficiency of the notable stochastic approximation method,average stochastic gradient descent(ASGD)in time-invariant systems,the authors propose an exponentially weighted moving average(EWMA)-based stochastic gradient descent(SGD)which accommodates the dynamic structure by introducing a forgetting factor and replacing the simple averaging step in ASGD with an EWMA step.Provided that the dynamic drift is Lipschitz continuous,the mean squared tracking error rate of the proposed method achieves the optimal rate in the nonparametric statistical paradigm.The proposed framework also allows us to derive the dynamic regret bound and asymptotic normality with a path variation constraint in a natural manner.Numerical analysis has been conducted to verify the performance of the proposed method.In particular,the proposed method is much more robust to the selection of learning rates compared with the ordinary SGD method.展开更多
This paper aims to investigate sign-based methods for unconstrained optimisation problems,which utilise the sign of gradients,instead of full gradients,to reduce the communication cost in distributed optimisation.The ...This paper aims to investigate sign-based methods for unconstrained optimisation problems,which utilise the sign of gradients,instead of full gradients,to reduce the communication cost in distributed optimisation.The methods introduced in this paper also provide an avenue to conquer the issue that classical sign gradient descent with a constant step size generally oscillates around the minimum value of the objective function.To this end,two sign-based algorithms are proposed and analysed.The first one,called ES-GD,is demonstrated to be linearly convergent to the optimal value if the objective function is strongly convex,smooth and separable.For the second variant,called ES-SGD,where the gradient estimates do not need to be unbiased,the algorithm is shown to converge linearly at the same rate as ES-GD to the optimal value under the same assumptions on the objective function.Numerical experiments are conducted to validate the theoretical results.展开更多
Optimization problems are prevalent in various fields,and the gradient-based gradient descent algorithm is a widely adopted optimization method.However,in classical computing,computing the numerical gradient for a fun...Optimization problems are prevalent in various fields,and the gradient-based gradient descent algorithm is a widely adopted optimization method.However,in classical computing,computing the numerical gradient for a function with variables necessitates at least d+1 function evaluations,resulting in a computational complexity of O(d).As the number of variables increases,the classical gradient estimation methods require substantial resources,ultimately surpassing the capabilities of classical computers.Fortunately,leveraging the principles of superposition and entanglement in quantum mechanics,quantum computers can achieve genuine parallel computing,leading to exponential acceleration over classical algorithms in some cases.In this paper,we propose a novel quantum-based gradient calculation method that requires only a single oracle calculation to obtain the numerical gradient result for a multivariate function.The complexity of this algorithm is just O(1).Building upon this approach,we successfully implemented the quantum gradient descent algorithm and applied it to the variational quantum eigensolver(VQE),creating a pure quantum variational optimization algorithm.Compared with classical gradient-based optimization algorithm,this quantum optimization algorithm has remarkable complexity advantages,providing an efficient solution to optimization problems.The proposed quantum-based method shows promise in enhancing the performance of optimization algorithms,highlighting the potential of quantum computing in this field.展开更多
Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing paramete...Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assumptions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.展开更多
The matrix rank minimization problem arises in many engineering applications. As this problem is NP-hard, a nonconvex relaxation of matrix rank minimization, called the Schatten-p quasi-norm minimization(0 < p <...The matrix rank minimization problem arises in many engineering applications. As this problem is NP-hard, a nonconvex relaxation of matrix rank minimization, called the Schatten-p quasi-norm minimization(0 < p < 1), has been developed to approximate the rank function closely. We study the performance of projected gradient descent algorithm for solving the Schatten-p quasi-norm minimization(0 < p < 1) problem.Based on the matrix restricted isometry property(M-RIP), we give the convergence guarantee and error bound for this algorithm and show that the algorithm is robust to noise with an exponential convergence rate.展开更多
The gradient descent approach is the key ingredient in variational quantum algorithms and machine learning tasks,which is an optimization algorithm for finding a local minimum of an objective function.The quantum vers...The gradient descent approach is the key ingredient in variational quantum algorithms and machine learning tasks,which is an optimization algorithm for finding a local minimum of an objective function.The quantum versions of gradient descent have been investigated and implemented in calculating molecular ground states and optimizing polynomial functions.Based on the quantum gradient descent algorithm and Choi-Jamiolkowski isomorphism,we present approaches to simulate efficiently the nonequilibrium steady states of Markovian open quantum many-body systems.Two strategies are developed to evaluate the expectation values of physical observables on the nonequilibrium steady states.Moreover,we adapt the quantum gradient descent algorithm to solve linear algebra problems including linear systems of equations and matrix-vector multiplications,by converting these algebraic problems into the simulations of closed quantum systems with well-defined Hamiltonians.Detailed examples are given to test numerically the effectiveness of the proposed algorithms for the dissipative quantum transverse Ising models and matrix-vector multiplications.展开更多
文摘Sentiment analysis plays an important role in distilling and clarifying content from movie reviews,aiding the audience in understanding universal views towards the movie.However,the abundance of reviews and the risk of encountering spoilers pose challenges for efcient sentiment analysis,particularly in Arabic content.Tis study proposed a Stochastic Gradient Descent(SGD)machine learning(ML)model tailored for sentiment analysis in Arabic and English movie reviews.SGD allows for fexible model complexity adjustments,which can adapt well to the Involvement of Arabic language data.Tis adaptability ensures that the model can capture the nuances and specifc local patterns of Arabic text,leading to better performance.Two distinct language datasets were utilized,and extensive pre-processing steps were employed to optimize the datasets for analysis.Te proposed SGD model,designed to accommodate the nuances of each language,aims to surpass existing models in terms of accuracy and efciency.Te SGD model achieves an accuracy of 84.89 on the Arabic dataset and 87.44 on the English dataset,making it the top-performing model in terms of accuracy on both datasets.Tis indicates that the SGD model consistently demonstrates high accuracy levels across Arabic and English datasets.Tis study helps deepen the understanding of sentiments across various linguistic datasets.Unlike many studies that focus solely on movie reviews,the Arabic dataset utilized here includes hotel reviews,ofering a broader perspective.
基金the Natural Science Foundation of Ningxia Province(No.2021AAC03230).
文摘Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.
基金the 2021 Key Project of Natural Science and Technology of Yangzhou Polytechnic Institute,Active Disturbance Rejection and Fault-Tolerant Control of Multi-Rotor Plant ProtectionUAV Based on QBall-X4(Grant Number 2021xjzk002).
文摘With the increasing prevalence of high-order systems in engineering applications, these systems often exhibitsignificant disturbances and can be challenging to model accurately. As a result, the active disturbance rejectioncontroller (ADRC) has been widely applied in various fields. However, in controlling plant protection unmannedaerial vehicles (UAVs), which are typically large and subject to significant disturbances, load disturbances andthe possibility of multiple actuator faults during pesticide spraying pose significant challenges. To address theseissues, this paper proposes a novel fault-tolerant control method that combines a radial basis function neuralnetwork (RBFNN) with a second-order ADRC and leverages a fractional gradient descent (FGD) algorithm.We integrate the plant protection UAV model’s uncertain parameters, load disturbance parameters, and actuatorfault parameters and utilize the RBFNN for system parameter identification. The resulting ADRC exhibits loaddisturbance suppression and fault tolerance capabilities, and our proposed active fault-tolerant control law hasLyapunov stability implications. Experimental results obtained using a multi-rotor fault-tolerant test platformdemonstrate that the proposed method outperforms other control strategies regarding load disturbance suppressionand fault-tolerant performance.
基金partially supported by NSF Grants DMS-1854434,DMS-1952644,and DMS-2151235 at UC Irvinesupported by NSF Grants DMS-1924935,DMS-1952339,DMS-2110145,DMS-2152762,and DMS-2208361,and DOE Grants DE-SC0021142 and DE-SC0002722.
文摘We prove,under mild conditions,the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model,both in batch gradient descent and stochastic gradient descent.We also discuss a Riemannian version of the Adam algorithm.We show numerical simulations of these algorithms on various benchmarks.
基金supported in part by the National Natural Science Foundation of China(61772493)the Deanship of Scientific Research(DSR)at King Abdulaziz University(RG-48-135-40)+1 种基金Guangdong Province Universities and College Pearl River Scholar Funded Scheme(2019)the Natural Science Foundation of Chongqing(cstc2019jcyjjqX0013)。
文摘A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.
基金Supported by the National Natural Science Foundation ofChina(No.61271240)Jiangsu Province Natural Science Fund Project(No.BK2010077)Subject of Twelfth Five Years Plans in Jiangsu Second Normal University(No.417103)
文摘Matrix completion is the extension of compressed sensing.In compressed sensing,we solve the underdetermined equations using sparsity prior of the unknown signals.However,in matrix completion,we solve the underdetermined equations based on sparsity prior in singular values set of the unknown matrix,which also calls low-rank prior of the unknown matrix.This paper firstly introduces basic concept of matrix completion,analyses the matrix suitably used in matrix completion,and shows that such matrix should satisfy two conditions:low rank and incoherence property.Then the paper provides three reconstruction algorithms commonly used in matrix completion:singular value thresholding algorithm,singular value projection,and atomic decomposition for minimum rank approximation,puts forward their shortcoming to know the rank of original matrix.The Projected Gradient Descent based on Soft Thresholding(STPGD),proposed in this paper predicts the rank of unknown matrix using soft thresholding,and iteratives based on projected gradient descent,thus it could estimate the rank of unknown matrix exactly with low computational complexity,this is verified by numerical experiments.We also analyze the convergence and computational complexity of the STPGD algorithm,point out this algorithm is guaranteed to converge,and analyse the number of iterations needed to reach reconstruction error.Compared the computational complexity of the STPGD algorithm to other algorithms,we draw the conclusion that the STPGD algorithm not only reduces the computational complexity,but also improves the precision of the reconstruction solution.
文摘In a wireless sensor network[1],the operation of a node depends on the battery power it carries.Because of the environmental reasons,the node cannot replace the battery.In order to improve the life cycle of the network,energy becomes one of the key problems in the design of the wireless sensor network(WSN)routing protocol[2].This paper proposes a routing protocol ERGD based on the method of gradient descent that can minimizes the consumption of energy.Within the communication radius of the current node,the distance between the current node and the next hop node is assumed that can generate a projected energy at the distance from the current node to the base station(BS),this projected energy and the remaining energy of the next hop node is the key factor in finding the next hop node.The simulation results show that the proposed protocol effectively extends the life cycle of the network and improves the reliability and fault tolerance of the system.
基金This project was supported by State Science &Technology Pursuing Project (2001BA204B01) of China and Foundation forUniversity Key Teacher by the Ministry of Education of China.
文摘The distribution of sampling data influences completeness of rule base so that extrapolating missing rules is very difficult. Based on data mining, a self-learning method is developed for identifying fuzzy model and extrapolating missing rules, by means of confidence measure and the improved gradient descent method. The proposed approach can not only identify fuzzy model, update its parameters and determine optimal output fuzzy sets simultaneously, but also resolve the uncontrollable problem led by the regions that data do not cover. The simulation results show the effectiveness and accuracy of the proposed approach with the classical truck backer-upper control problem verifying.
基金supported by the Fundamental Research Funds for the Central Universities(No.56XAA17075)
文摘A gradient descent algorithm with adjustable parameter for attitude estimation is developed,aiming at the attitude measurement for small unmanned aerial vehicle(UAV)in real-time flight conditions.The accelerometer and magnetometer are introduced to construct an error equation with the gyros,thus the drifting characteristics of gyroscope can be compensated by solving the error equation utilized by the gradient descent algorithm.Performance of the presented algorithm is evaluated using a self-proposed micro-electro-mechanical system(MEMS)based attitude heading reference system which is mounted on a tri-axis turntable.The on-ground,turntable and flight experiments indicate that the estimation attitude has a good accuracy.Also,the presented system is compared with an open-source flight control system which runs extended Kalman filter(EKF),and the results show that the attitude control system using the gradient descent method can estimate the attitudes for UAV effectively.
文摘Motion estimation is an important part of the MPEG- 4 encoder, due to its significant impact on the bit rate and the output quality of the encoder sequence. Unfortunately this feature takes a significant part of the encoding time especially when the straightforward full search(FS) algorithm is used. In this paper, a new algorithm named diamond block based gradient descent search (DBBGDS) algorithm, which is significantly faster than FS and gives similar quality of the output sequence, is proposed. At the same time, some other algorithms, such as three step search (TSS), improved three step search (ITSS), new three step search (NTSS), four step search (4SS), cellular search (CS) , diamond search (DS) and block based gradient descent search (BBGDS), are adopted and compared with DBBGDS. As the experimental results show, DBBGDS has its own advantages. Although DS has been adopted by the MPEG- 4 VM, its output sequence quality is worse than that of the proposed algorithm while its complexity is similar to the proposed one. Compared with BBGDS, the proposed algorithm can achieve a better output quality.
基金funded by State Key Laboratory for GeoMechanics and Deep Underground Engineering&Institute for Deep Underground Science and Engineering,Grant Number XD2021021BUCEA Post Graduate Innovation Project under Grant,Grant Number PG2023092.
文摘Rockburst is a phenomenon in which free surfaces are formed during excavation,which subsequently causes the sudden release of energy in the construction of mines and tunnels.Light rockburst only peels off rock slices without ejection,while severe rockburst causes casualties and property loss.The frequency and degree of rockburst damage increases with the excavation depth.Moreover,rockburst is the leading engineering geological hazard in the excavation process,and thus the prediction of its intensity grade is of great significance to the development of geotechnical engineering.Therefore,the prediction of rockburst intensity grade is one problem that needs to be solved urgently.By comprehensively considering the occurrence mechanism of rockburst,this paper selects the stress index(σθ/σc),brittleness index(σ_(c)/σ_(t)),and rock elastic energy index(Wet)as the rockburst evaluation indexes through the Spearman coefficient method.This overcomes the low accuracy problem of a single evaluation index prediction method.Following this,the BGD-MSR-DNN rockburst intensity grade prediction model based on batch gradient descent and a multi-scale residual deep neural network is proposed.The batch gradient descent(BGD)module is used to replace the gradient descent algorithm,which effectively improves the efficiency of the network and reduces the model training time.Moreover,the multi-scale residual(MSR)module solves the problem of network degradation when there are too many hidden layers of the deep neural network(DNN),thus improving the model prediction accuracy.The experimental results reveal the BGDMSR-DNN model accuracy to reach 97.1%,outperforming other comparable models.Finally,actual projects such as Qinling Tunnel and Daxiangling Tunnel,reached an accuracy of 100%.The model can be applied in mines and tunnel engineering to realize the accurate and rapid prediction of rockburst intensity grade.
基金Open Foundation of State Key Lab of Transmission of Wide-Band FiberTechnologies of Communication Systems
文摘A new algorithm to exploit the learning rates of gradient descent method is presented, based on the second-order Taylor expansion of the error energy function with respect to learning rate, at some values decided by "award-punish" strategy. Detailed deduction of the algorithm applied to RBF networks is given. Simulation studies show that this algorithm can increase the rate of convergence and improve the performance of the gradient descent method.
文摘Regulating the power output for a power plant as demand for electricity fluctuates throughout the day is important for both economic purpose and the safety of the generator. In this work, gradient descent method together with regularization is investigated to study the electricity output related to vacuum level and temperature in the turbine. Ninety percent of the data was used to train the regression parameters while the remaining ten percent was used for validation. Final results showed that 99% accuracy could be obtained with this method. This opens a new window for electricity output prediction for power plants.
基金first author is partially supported by the University of Kashan(1143902/2).
文摘In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(1/√T) convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the FashionMinst, CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic step size improves test accuracy by 0.9% for the CIFAR100 dataset when we utilize a convolutional neural network (CNN) model.
基金supported by the National Key R&D Program of China under Grant Nos.2022YFA1003703,2022YFA1003800the Natural Science Foundation of Tianjin under Grant Nos.23JCQNJC01410the National Natural Science Foundation of China under Grant Nos.11925106,12231011,11931001,12101332,12226007,12326325,and 12471260。
文摘In many applications involving data streams,the sequences of data arise from highly dynamic and often unstable real-life processes,rendering untenable the standard assumption that current and future data come from the same distribution.In response,new methodologies,such as dynamic online learning,have been proposed in order to account for the nonstationary features in the datagenerating process.Motivated by the stability and statistical efficiency of the notable stochastic approximation method,average stochastic gradient descent(ASGD)in time-invariant systems,the authors propose an exponentially weighted moving average(EWMA)-based stochastic gradient descent(SGD)which accommodates the dynamic structure by introducing a forgetting factor and replacing the simple averaging step in ASGD with an EWMA step.Provided that the dynamic drift is Lipschitz continuous,the mean squared tracking error rate of the proposed method achieves the optimal rate in the nonparametric statistical paradigm.The proposed framework also allows us to derive the dynamic regret bound and asymptotic normality with a path variation constraint in a natural manner.Numerical analysis has been conducted to verify the performance of the proposed method.In particular,the proposed method is much more robust to the selection of learning rates compared with the ordinary SGD method.
基金supported by the National Natural Science Foundation of China under Grants 62473292 and 62088101.
文摘This paper aims to investigate sign-based methods for unconstrained optimisation problems,which utilise the sign of gradients,instead of full gradients,to reduce the communication cost in distributed optimisation.The methods introduced in this paper also provide an avenue to conquer the issue that classical sign gradient descent with a constant step size generally oscillates around the minimum value of the objective function.To this end,two sign-based algorithms are proposed and analysed.The first one,called ES-GD,is demonstrated to be linearly convergent to the optimal value if the objective function is strongly convex,smooth and separable.For the second variant,called ES-SGD,where the gradient estimates do not need to be unbiased,the algorithm is shown to converge linearly at the same rate as ES-GD to the optimal value under the same assumptions on the objective function.Numerical experiments are conducted to validate the theoretical results.
基金supported by the National Natural Science Foundation of China under Grant No.12105195.
文摘Optimization problems are prevalent in various fields,and the gradient-based gradient descent algorithm is a widely adopted optimization method.However,in classical computing,computing the numerical gradient for a function with variables necessitates at least d+1 function evaluations,resulting in a computational complexity of O(d).As the number of variables increases,the classical gradient estimation methods require substantial resources,ultimately surpassing the capabilities of classical computers.Fortunately,leveraging the principles of superposition and entanglement in quantum mechanics,quantum computers can achieve genuine parallel computing,leading to exponential acceleration over classical algorithms in some cases.In this paper,we propose a novel quantum-based gradient calculation method that requires only a single oracle calculation to obtain the numerical gradient result for a multivariate function.The complexity of this algorithm is just O(1).Building upon this approach,we successfully implemented the quantum gradient descent algorithm and applied it to the variational quantum eigensolver(VQE),creating a pure quantum variational optimization algorithm.Compared with classical gradient-based optimization algorithm,this quantum optimization algorithm has remarkable complexity advantages,providing an efficient solution to optimization problems.The proposed quantum-based method shows promise in enhancing the performance of optimization algorithms,highlighting the potential of quantum computing in this field.
基金supported by the National Natural Science Foundation of China(Nos.11731013,U19B2040,11991022)by the Leading Project of the Chinese Academy of Sciences(Nos.XDA27010102,XDA27010302)。
文摘Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assumptions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.
基金supported by National Natural Science Foundation of China(Grant No.11171299)
文摘The matrix rank minimization problem arises in many engineering applications. As this problem is NP-hard, a nonconvex relaxation of matrix rank minimization, called the Schatten-p quasi-norm minimization(0 < p < 1), has been developed to approximate the rank function closely. We study the performance of projected gradient descent algorithm for solving the Schatten-p quasi-norm minimization(0 < p < 1) problem.Based on the matrix restricted isometry property(M-RIP), we give the convergence guarantee and error bound for this algorithm and show that the algorithm is robust to noise with an exponential convergence rate.
基金supported by the National Natural Science Foundation of China(Grant Nos.12075159,12171044,and 12005015)Beijing Natural Science Foundation(Grant No.Z190005)Academy for Multidisciplinary Studies,Capital Normal University,Academician Innovation Platform of Hainan Province,and Shenzhen Institute for Quantum Science and Engineering,Southern University of Science and Technology(Grant No.SIQSE202001)。
文摘The gradient descent approach is the key ingredient in variational quantum algorithms and machine learning tasks,which is an optimization algorithm for finding a local minimum of an objective function.The quantum versions of gradient descent have been investigated and implemented in calculating molecular ground states and optimizing polynomial functions.Based on the quantum gradient descent algorithm and Choi-Jamiolkowski isomorphism,we present approaches to simulate efficiently the nonequilibrium steady states of Markovian open quantum many-body systems.Two strategies are developed to evaluate the expectation values of physical observables on the nonequilibrium steady states.Moreover,we adapt the quantum gradient descent algorithm to solve linear algebra problems including linear systems of equations and matrix-vector multiplications,by converting these algebraic problems into the simulations of closed quantum systems with well-defined Hamiltonians.Detailed examples are given to test numerically the effectiveness of the proposed algorithms for the dissipative quantum transverse Ising models and matrix-vector multiplications.
