As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dim...As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.展开更多
In edge computing,though distributed training of Deep Neural Networks(DNNs)is expected to exchange massive gradients between parameter servers and working nodes,the high communication cost constrains the training spee...In edge computing,though distributed training of Deep Neural Networks(DNNs)is expected to exchange massive gradients between parameter servers and working nodes,the high communication cost constrains the training speed.To break this limitation,gradient compression algorithms expect the ultimate compression ratio at the expense of the accuracy of the trained model.Therefore,new gradient compression techniques are necessary to ensure both communication efficiency and model accuracy.This paper introduces a novel technique—an Adaptive Sparse Ternary Gradient Compression(ASTC)scheme,which relies on the number of gradients in model layers to compress gradients.ASTC establishes the model compression selection criterion by gradients’amount,compresses the network layer that meets the model’s standard,evaluates the gradients’importance based on entropy to adaptively perform sparse compression,and finally conducts ternary quantization compression and a lossless code scheme on sparse gradients.Using public datasets(MNIST,CIFAR-10,Tiny ImageNet)and deep learning models(CNN,LeNet5,ResNet18)for experimental evaluation,we exhibit excellent results that the training efficiency of ASTC is about 1.6 times,1.37 times,and 1.1 times higher than that of Top-1,AdaComp,and SBC,respectively.Furthermore,ASTC can be improved by an average of about 1.9%in training accuracy compared with the above approaches.展开更多
The federated self-supervised framework is a distributed machine learning method that combines federated learning and self-supervised learning, which can effectively solve the problem of traditional federated learning...The federated self-supervised framework is a distributed machine learning method that combines federated learning and self-supervised learning, which can effectively solve the problem of traditional federated learning being difficult to process large-scale unlabeled data. The existing federated self-supervision framework has problems with low communication efficiency and high communication delay between clients and central servers. Therefore, we added edge servers to the federated self-supervision framework to reduce the pressure on the central server caused by frequent communication between both ends. A communication compression scheme using gradient quantization and sparsification was proposed to optimize the communication of the entire framework, and the algorithm of the sparse communication compression module was improved. Experiments have proved that the learning rate changes of the improved sparse communication compression module are smoother and more stable. Our communication compression scheme effectively reduced the overall communication overhead.展开更多
Image semantic segmentation has become an essential part of autonomous driving.To further improve the generalization ability and the robustness of semantic segmentation algorithms,a lightweight algorithm network based...Image semantic segmentation has become an essential part of autonomous driving.To further improve the generalization ability and the robustness of semantic segmentation algorithms,a lightweight algorithm network based on Squeeze-and-Excitation Attention Mechanism(SE)and Depthwise Separable Convolution(DSC)is designed.Meanwhile,Adam-GC,an Adam optimization algorithm based on Gradient Compression(GC),is proposed to improve the training speed,segmentation accuracy,generalization ability and stability of the algorithm network.To verify and compare the effectiveness of the algorithm network proposed in this paper,the trained networkmodel is used for experimental verification and comparative test on the Cityscapes semantic segmentation dataset.The validation and comparison results show that the overall segmentation results of the algorithmnetwork can achieve 78.02%MIoU on Cityscapes validation set,which is better than the basic algorithm network and the other latest semantic segmentation algorithms network.Besides meeting the stability and accuracy requirements,it has a particular significance for the development of image semantic segmentation.展开更多
基金supported in part by the National Key Research and Development Program of China under Grant 2020YFB1807700in part by the National Science Foundation of China under Grant U200120122
文摘As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.
基金supported by the National Natural Science Foundation of China(Grant No.61902110,61832005)the Key Research and Development Program of China,Jiangsu Province(BE2020729)+1 种基金the Transformation Program of Scientific and Technological Achievements of Jiangsu Provence(BA2021002)the Key Technology Project of China Huaneng Group(Grant No.HNKJ20-H64,HNKJ19-H12).
文摘In edge computing,though distributed training of Deep Neural Networks(DNNs)is expected to exchange massive gradients between parameter servers and working nodes,the high communication cost constrains the training speed.To break this limitation,gradient compression algorithms expect the ultimate compression ratio at the expense of the accuracy of the trained model.Therefore,new gradient compression techniques are necessary to ensure both communication efficiency and model accuracy.This paper introduces a novel technique—an Adaptive Sparse Ternary Gradient Compression(ASTC)scheme,which relies on the number of gradients in model layers to compress gradients.ASTC establishes the model compression selection criterion by gradients’amount,compresses the network layer that meets the model’s standard,evaluates the gradients’importance based on entropy to adaptively perform sparse compression,and finally conducts ternary quantization compression and a lossless code scheme on sparse gradients.Using public datasets(MNIST,CIFAR-10,Tiny ImageNet)and deep learning models(CNN,LeNet5,ResNet18)for experimental evaluation,we exhibit excellent results that the training efficiency of ASTC is about 1.6 times,1.37 times,and 1.1 times higher than that of Top-1,AdaComp,and SBC,respectively.Furthermore,ASTC can be improved by an average of about 1.9%in training accuracy compared with the above approaches.
文摘The federated self-supervised framework is a distributed machine learning method that combines federated learning and self-supervised learning, which can effectively solve the problem of traditional federated learning being difficult to process large-scale unlabeled data. The existing federated self-supervision framework has problems with low communication efficiency and high communication delay between clients and central servers. Therefore, we added edge servers to the federated self-supervision framework to reduce the pressure on the central server caused by frequent communication between both ends. A communication compression scheme using gradient quantization and sparsification was proposed to optimize the communication of the entire framework, and the algorithm of the sparse communication compression module was improved. Experiments have proved that the learning rate changes of the improved sparse communication compression module are smoother and more stable. Our communication compression scheme effectively reduced the overall communication overhead.
基金supported by Qingdao People’s Livelihood Science and Technology Plan (Grant 19-6-1-88-nsh).
文摘Image semantic segmentation has become an essential part of autonomous driving.To further improve the generalization ability and the robustness of semantic segmentation algorithms,a lightweight algorithm network based on Squeeze-and-Excitation Attention Mechanism(SE)and Depthwise Separable Convolution(DSC)is designed.Meanwhile,Adam-GC,an Adam optimization algorithm based on Gradient Compression(GC),is proposed to improve the training speed,segmentation accuracy,generalization ability and stability of the algorithm network.To verify and compare the effectiveness of the algorithm network proposed in this paper,the trained networkmodel is used for experimental verification and comparative test on the Cityscapes semantic segmentation dataset.The validation and comparison results show that the overall segmentation results of the algorithmnetwork can achieve 78.02%MIoU on Cityscapes validation set,which is better than the basic algorithm network and the other latest semantic segmentation algorithms network.Besides meeting the stability and accuracy requirements,it has a particular significance for the development of image semantic segmentation.
