Sentiment analysis plays an important role in distilling and clarifying content from movie reviews,aiding the audience in understanding universal views towards the movie.However,the abundance of reviews and the risk o...Sentiment analysis plays an important role in distilling and clarifying content from movie reviews,aiding the audience in understanding universal views towards the movie.However,the abundance of reviews and the risk of encountering spoilers pose challenges for efcient sentiment analysis,particularly in Arabic content.Tis study proposed a Stochastic Gradient Descent(SGD)machine learning(ML)model tailored for sentiment analysis in Arabic and English movie reviews.SGD allows for fexible model complexity adjustments,which can adapt well to the Involvement of Arabic language data.Tis adaptability ensures that the model can capture the nuances and specifc local patterns of Arabic text,leading to better performance.Two distinct language datasets were utilized,and extensive pre-processing steps were employed to optimize the datasets for analysis.Te proposed SGD model,designed to accommodate the nuances of each language,aims to surpass existing models in terms of accuracy and efciency.Te SGD model achieves an accuracy of 84.89 on the Arabic dataset and 87.44 on the English dataset,making it the top-performing model in terms of accuracy on both datasets.Tis indicates that the SGD model consistently demonstrates high accuracy levels across Arabic and English datasets.Tis study helps deepen the understanding of sentiments across various linguistic datasets.Unlike many studies that focus solely on movie reviews,the Arabic dataset utilized here includes hotel reviews,ofering a broader perspective.展开更多
A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and...A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.展开更多
Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for ga...Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.展开更多
In this paper,we consider a distributed resource allocation problem of minimizing a global convex function formed by a sum of local convex functions with coupling constraints.Based on neighbor communication and stocha...In this paper,we consider a distributed resource allocation problem of minimizing a global convex function formed by a sum of local convex functions with coupling constraints.Based on neighbor communication and stochastic gradient,a distributed stochastic mirror descent algorithm is designed for the distributed resource allocation problem.Sublinear convergence to an optimal solution of the proposed algorithm is given when the second moments of the gradient noises are summable.A numerical example is also given to illustrate the effectiveness of the proposed algorithm.展开更多
As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dim...As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.展开更多
In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(1/√T) convergence ...In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(1/√T) convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the FashionMinst, CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic step size improves test accuracy by 0.9% for the CIFAR100 dataset when we utilize a convolutional neural network (CNN) model.展开更多
Stochastic gradient descent(SGD)methods have gained widespread popularity for solving large-scale optimization problems.However,the inherent variance in SGD often leads to slow convergence rates.We introduce a family ...Stochastic gradient descent(SGD)methods have gained widespread popularity for solving large-scale optimization problems.However,the inherent variance in SGD often leads to slow convergence rates.We introduce a family of unbiased stochastic gradient estimators that encompasses existing estimators from the literature and identify a gradient estimator that not only maintains unbiasedness but also achieves minimal variance.Compared with the existing estimator used in SGD algorithms,the proposed estimator demonstrates a significant reduction in variance.By utilizing this stochastic gradient estimator to approximate the full gradient,we propose two mini-batch stochastic conjugate gradient algorithms with minimal variance.Under the assumptions of strong convexity and smoothness on the objective function,we prove that the two algorithms achieve linear convergence rates.Numerical experiments validate the effectiveness of the proposed gradient estimator in reducing variance and demonstrate that the two stochastic conjugate gradient algorithms exhibit accelerated convergence rates and enhanced stability.展开更多
Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing paramete...Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assumptions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.展开更多
In this work,we develop a stochastic gradient descent method for the computational optimal design of random rough surfaces in thin-film solar cells.We formulate the design problems as random PDE-constrained optimizati...In this work,we develop a stochastic gradient descent method for the computational optimal design of random rough surfaces in thin-film solar cells.We formulate the design problems as random PDE-constrained optimization problems and seek the optimal statistical parameters for the random surfaces.The optimizations at fixed frequency as well as at multiple frequencies and multiple incident angles are investigated.To evaluate the gradient of the objective function,we derive the shape derivatives for the interfaces and apply the adjoint state method to perform the computation.The stochastic gradient descent method evaluates the gradient of the objective function only at a few samples for each iteration,which reduces the computational cost significantly.Various numerical experiments are conducted to illustrate the efficiency of the method and significant increases of the absorptance for the optimal random structures.We also examine the convergence of the stochastic gradient descent algorithm theoretically and prove that the numerical method is convergent under certain assumptions for the random interfaces.展开更多
为了保障网络环境的安全性,提出了基于集成式图卷积神经网络算法的网络入侵检测技术。研究方法采用随机梯度下降算法和均方根传播(Root Mean Square Propagation,RMSProp)优化器提升了检测模型的训练效率,强化了检测模型的分类效果。研...为了保障网络环境的安全性,提出了基于集成式图卷积神经网络算法的网络入侵检测技术。研究方法采用随机梯度下降算法和均方根传播(Root Mean Square Propagation,RMSProp)优化器提升了检测模型的训练效率,强化了检测模型的分类效果。研究结果显示,研究模型的入侵检测准确率为96.41%~97.18%。可见经过研究模型优化后,入侵检测技术在模型训练效率和模型训练精度上都有明显提升。研究模型可以根据访问来源进行数据分类,提升了入侵检测模型对访问行为的分类效果。同时,分类效果的提升优化了计算机对攻击行为的识别效率,使计算机的防御效果增强,有效保障了用户的网络安全环境。因此,研究为网络入侵行为的检测提供了一个识别效果较好的技术方法。展开更多
为了减少深度学习中隐私泄露的风险,许多研究利用差分隐私技术来训练神经网络.然而,这些隐私保护方法通常会导致模型性能显著下降.为了在隐私保护与模型效用之间实现平衡,本文提出了一种基于稀疏平滑自蒸馏的差分隐私深度学习(Different...为了减少深度学习中隐私泄露的风险,许多研究利用差分隐私技术来训练神经网络.然而,这些隐私保护方法通常会导致模型性能显著下降.为了在隐私保护与模型效用之间实现平衡,本文提出了一种基于稀疏平滑自蒸馏的差分隐私深度学习(Differentially Private learning with sparse and smooth Self-Distillation,DP3SD)方法,通过双温度缩放机制来增强隐私保护深度学习的效用.具体而言,该方法设计了一种由稀疏分类损失和光滑蒸馏损失组成的双温度缩放损失函数.通过将较低温度应用于分类损失,能够使学生模型的类别预测分布更加锐化,从而减少低概率类别的影响,这些类别通常可能是由噪声引起的.相反,较高温度应用于蒸馏损失,能够平滑教师模型和学生模型的预测分布,从而在差分隐私约束下实现稳定和高效的知识迁移.在差分隐私随机梯度下降的严格隐私保障下,本文提出的双重缩放机制能够减轻噪声带来的扰动,提升学生模型的泛化能力.在三个公开数据集上的大量实验表明:本文提出的方法能够在确保严格数据隐私的同时,增强模型的可用性.展开更多
文摘Sentiment analysis plays an important role in distilling and clarifying content from movie reviews,aiding the audience in understanding universal views towards the movie.However,the abundance of reviews and the risk of encountering spoilers pose challenges for efcient sentiment analysis,particularly in Arabic content.Tis study proposed a Stochastic Gradient Descent(SGD)machine learning(ML)model tailored for sentiment analysis in Arabic and English movie reviews.SGD allows for fexible model complexity adjustments,which can adapt well to the Involvement of Arabic language data.Tis adaptability ensures that the model can capture the nuances and specifc local patterns of Arabic text,leading to better performance.Two distinct language datasets were utilized,and extensive pre-processing steps were employed to optimize the datasets for analysis.Te proposed SGD model,designed to accommodate the nuances of each language,aims to surpass existing models in terms of accuracy and efciency.Te SGD model achieves an accuracy of 84.89 on the Arabic dataset and 87.44 on the English dataset,making it the top-performing model in terms of accuracy on both datasets.Tis indicates that the SGD model consistently demonstrates high accuracy levels across Arabic and English datasets.Tis study helps deepen the understanding of sentiments across various linguistic datasets.Unlike many studies that focus solely on movie reviews,the Arabic dataset utilized here includes hotel reviews,ofering a broader perspective.
基金supported in part by the National Natural Science Foundation of China(61772493)the Deanship of Scientific Research(DSR)at King Abdulaziz University(RG-48-135-40)+1 种基金Guangdong Province Universities and College Pearl River Scholar Funded Scheme(2019)the Natural Science Foundation of Chongqing(cstc2019jcyjjqX0013)。
文摘A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.
基金the Natural Science Foundation of Ningxia Province(No.2021AAC03230).
文摘Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.
基金the National Key Research and Development Program of China(No.2016YFB0901900)the National Natural Science Foundation of China(No.61733018)the China Special Postdoctoral Science Foundation Funded Project(No.Y990075G21).
文摘In this paper,we consider a distributed resource allocation problem of minimizing a global convex function formed by a sum of local convex functions with coupling constraints.Based on neighbor communication and stochastic gradient,a distributed stochastic mirror descent algorithm is designed for the distributed resource allocation problem.Sublinear convergence to an optimal solution of the proposed algorithm is given when the second moments of the gradient noises are summable.A numerical example is also given to illustrate the effectiveness of the proposed algorithm.
基金supported in part by the National Key Research and Development Program of China under Grant 2020YFB1807700in part by the National Science Foundation of China under Grant U200120122
文摘As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.
基金first author is partially supported by the University of Kashan(1143902/2).
文摘In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an O(1/√T) convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the FashionMinst, CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic step size improves test accuracy by 0.9% for the CIFAR100 dataset when we utilize a convolutional neural network (CNN) model.
基金supported by the Strategic Priority Research Program of Chinese Academy of Sciences(Grant No.XDA27010101)the Beijing Natural Science Foundation(Grant No.Z220004)+1 种基金the Chinese NSF(Grant No.12021001)the Fundamental Research Funds for the Central Universities(Grant No.2023ZCJH02)。
文摘Stochastic gradient descent(SGD)methods have gained widespread popularity for solving large-scale optimization problems.However,the inherent variance in SGD often leads to slow convergence rates.We introduce a family of unbiased stochastic gradient estimators that encompasses existing estimators from the literature and identify a gradient estimator that not only maintains unbiasedness but also achieves minimal variance.Compared with the existing estimator used in SGD algorithms,the proposed estimator demonstrates a significant reduction in variance.By utilizing this stochastic gradient estimator to approximate the full gradient,we propose two mini-batch stochastic conjugate gradient algorithms with minimal variance.Under the assumptions of strong convexity and smoothness on the objective function,we prove that the two algorithms achieve linear convergence rates.Numerical experiments validate the effectiveness of the proposed gradient estimator in reducing variance and demonstrate that the two stochastic conjugate gradient algorithms exhibit accelerated convergence rates and enhanced stability.
基金supported by the National Natural Science Foundation of China(Nos.11731013,U19B2040,11991022)by the Leading Project of the Chinese Academy of Sciences(Nos.XDA27010102,XDA27010302)。
文摘Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assumptions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.
基金partially supported by the DOE grant DE-SC0022253the work of JL was partially supported by the NSF grant DMS-1719851 and DMS-2011148.
文摘In this work,we develop a stochastic gradient descent method for the computational optimal design of random rough surfaces in thin-film solar cells.We formulate the design problems as random PDE-constrained optimization problems and seek the optimal statistical parameters for the random surfaces.The optimizations at fixed frequency as well as at multiple frequencies and multiple incident angles are investigated.To evaluate the gradient of the objective function,we derive the shape derivatives for the interfaces and apply the adjoint state method to perform the computation.The stochastic gradient descent method evaluates the gradient of the objective function only at a few samples for each iteration,which reduces the computational cost significantly.Various numerical experiments are conducted to illustrate the efficiency of the method and significant increases of the absorptance for the optimal random structures.We also examine the convergence of the stochastic gradient descent algorithm theoretically and prove that the numerical method is convergent under certain assumptions for the random interfaces.
文摘为了保障网络环境的安全性,提出了基于集成式图卷积神经网络算法的网络入侵检测技术。研究方法采用随机梯度下降算法和均方根传播(Root Mean Square Propagation,RMSProp)优化器提升了检测模型的训练效率,强化了检测模型的分类效果。研究结果显示,研究模型的入侵检测准确率为96.41%~97.18%。可见经过研究模型优化后,入侵检测技术在模型训练效率和模型训练精度上都有明显提升。研究模型可以根据访问来源进行数据分类,提升了入侵检测模型对访问行为的分类效果。同时,分类效果的提升优化了计算机对攻击行为的识别效率,使计算机的防御效果增强,有效保障了用户的网络安全环境。因此,研究为网络入侵行为的检测提供了一个识别效果较好的技术方法。
文摘为了减少深度学习中隐私泄露的风险,许多研究利用差分隐私技术来训练神经网络.然而,这些隐私保护方法通常会导致模型性能显著下降.为了在隐私保护与模型效用之间实现平衡,本文提出了一种基于稀疏平滑自蒸馏的差分隐私深度学习(Differentially Private learning with sparse and smooth Self-Distillation,DP3SD)方法,通过双温度缩放机制来增强隐私保护深度学习的效用.具体而言,该方法设计了一种由稀疏分类损失和光滑蒸馏损失组成的双温度缩放损失函数.通过将较低温度应用于分类损失,能够使学生模型的类别预测分布更加锐化,从而减少低概率类别的影响,这些类别通常可能是由噪声引起的.相反,较高温度应用于蒸馏损失,能够平滑教师模型和学生模型的预测分布,从而在差分隐私约束下实现稳定和高效的知识迁移.在差分隐私随机梯度下降的严格隐私保障下,本文提出的双重缩放机制能够减轻噪声带来的扰动,提升学生模型的泛化能力.在三个公开数据集上的大量实验表明:本文提出的方法能够在确保严格数据隐私的同时,增强模型的可用性.
