Federated learning often experiences slow and unstable convergence due to edge-side data heterogeneity.This problem becomes more severe when edge participation rate is low,as the information collected from different e...Federated learning often experiences slow and unstable convergence due to edge-side data heterogeneity.This problem becomes more severe when edge participation rate is low,as the information collected from different edge devices varies significantly.As a result,communication overhead increases,which further slows down the convergence process.To address this challenge,we propose a simple yet effective federated learning framework that improves consistency among edge devices.The core idea is clusters the lookahead gradients collected from edge devices on the cloud server to obtain personalized momentum for steering local updates.In parallel,a global momentum is applied during model aggregation,enabling faster convergence while preserving personalization.This strategy enables efficient propagation of the estimated global update direction to all participating edge devices and maintains alignment in local training,without introducing extra memory or communication overhead.We conduct extensive experiments on benchmark datasets such as Cifar100 and Tiny-ImageNet.The results confirm the effectiveness of our framework.On CIFAR-100,our method reaches 55%accuracy with 37 fewer rounds and achieves a competitive final accuracy of 65.46%.Even under extreme non-IID scenarios,it delivers significant improvements in both accuracy and communication efficiency.The implementation is publicly available at https://github.com/sjmp525/CollaborativeComputing/tree/FedCCM(accessed on 20 October 2025).展开更多
联邦学习因具有隐私保护的天然特性,已经逐渐成为一个被广泛认可的分布式机器学习框架。但由于参与方数据分布的差异性,特别是呈现非独立同分布(Non-Independent and Identically Distributed,Non-IID)时,其面临着泛化性能不足、收敛性...联邦学习因具有隐私保护的天然特性,已经逐渐成为一个被广泛认可的分布式机器学习框架。但由于参与方数据分布的差异性,特别是呈现非独立同分布(Non-Independent and Identically Distributed,Non-IID)时,其面临着泛化性能不足、收敛性能下降、数据倾斜等严峻挑战。用预训练基础模型缓解Non-IID问题作为一种新颖的方法,演变出了各种各样的解决方案。对此,从预训练基础模型的角度,对现有工作进行了综述。首先介绍了基础模型方法,对典型的基础模型编码结构进行对比分析。其次从修改输入、基础模型部分结构再训练,以及参数高效微调3个角度,提出了一种新的分类方法。最后探讨了该类工作的核心难题和未来研究方向。展开更多
Sharing data while protecting privacy in the industrial Internet is a significant challenge.Traditional machine learning methods require a combination of all data for training;however,this approach can be limited by d...Sharing data while protecting privacy in the industrial Internet is a significant challenge.Traditional machine learning methods require a combination of all data for training;however,this approach can be limited by data availability and privacy concerns.Federated learning(FL)has gained considerable attention because it allows for decentralized training on multiple local datasets.However,the training data collected by data providers are often non-independent and identically distributed(non-IID),resulting in poor FL performance.This paper proposes a privacy-preserving approach for sharing non-IID data in the industrial Internet using an FL approach based on blockchain technology.To overcome the problem of non-IID data leading to poor training accuracy,we propose dynamically updating the local model based on the divergence of the global and local models.This approach can significantly improve the accuracy of FL training when there is relatively large dispersion.In addition,we design a dynamic gradient clipping algorithm to alleviate the influence of noise on the model accuracy to reduce potential privacy leakage caused by sharing model parameters.Finally,we evaluate the performance of the proposed scheme using commonly used open-source image datasets.The simulation results demonstrate that our method can significantly enhance the accuracy while protecting privacy and maintaining efficiency,thereby providing a new solution to data-sharing and privacy-protection challenges in the industrial Internet.展开更多
Federated learning(FL) is a machine learning paradigm for data silos and privacy protection,which aims to organize multiple clients for training global machine learning models without exposing data to all parties.Howe...Federated learning(FL) is a machine learning paradigm for data silos and privacy protection,which aims to organize multiple clients for training global machine learning models without exposing data to all parties.However,when dealing with non-independently identically distributed(non-ⅡD) client data,FL cannot obtain more satisfactory results than centrally trained machine learning and even fails to match the accuracy of the local model obtained by client training alone.To analyze and address the above issues,we survey the state-of-theart methods in the literature related to FL on non-ⅡD data.On this basis,a motivation-based taxonomy,which classifies these methods into two categories,including heterogeneity reducing strategies and adaptability enhancing strategies,is proposed.Moreover,the core ideas and main challenges of these methods are analyzed.Finally,we envision several promising research directions that have not been thoroughly studied,in hope of promoting research in related fields to a certain extent.展开更多
In non-independent and identically distributed(non-IID)data environments,model performance often degrades significantly.To address this issue,two improvement methods are proposed:FedReg and FedReg^(*).FedReg is a meth...In non-independent and identically distributed(non-IID)data environments,model performance often degrades significantly.To address this issue,two improvement methods are proposed:FedReg and FedReg^(*).FedReg is a method based on hybrid regularization aimed at enhancing federated learning in non-IID scenarios.It introduces hybrid regularization to replace traditional L2 regularization,combining the advantages of L1 and L2 regularization to enable feature selection while preventing overfitting.This method better adapts to the diverse data distributions of different clients,improving the overall model performance.FedReg^(*)combines hybrid regularization with weighted model aggregation.In addition to the benefits of hybrid regularization,FedReg^(*)applies a weighted averaging method in the model aggregation process,calculating weights based on the cosine similarity between each client gradient and the global gradient to more reasonably distribute client contributions.By considering variations in data quality and quantity among clients,FedReg^(*)highlights the importance of key clients and enhances the model’s generalization performance.These improvement methods enhance model accuracy and communication efficiency.展开更多
With the increasing complexity of vehicular networks and the proliferation of connected vehicles,Federated Learning(FL)has emerged as a critical framework for decentralized model training while preserving data privacy...With the increasing complexity of vehicular networks and the proliferation of connected vehicles,Federated Learning(FL)has emerged as a critical framework for decentralized model training while preserving data privacy.However,efficient client selection and adaptive weight allocation in heterogeneous and non-IID environments remain challenging.To address these issues,we propose Federated Learning with Client Selection and Adaptive Weighting(FedCW),a novel algorithm that leverages adaptive client selection and dynamic weight allocation for optimizing model convergence in real-time vehicular networks.FedCW selects clients based on their Euclidean distance from the global model and dynamically adjusts aggregation weights to optimize both data diversity and model convergence.Experimental results show that FedCW significantly outperforms existing FL algorithms such as FedAvg,FedProx,and SCAFFOLD,particularly in non-IID settings,achieving faster convergence,higher accuracy,and reduced communication overhead.These findings demonstrate that FedCW provides an effective solution for enhancing the performance of FL in heterogeneous,edge-based computing environments.展开更多
Distributed stochastic gradient descent and its variants have been widely adopted in the training of machine learning models,which apply multiple workers in parallel.Among them,local-based algorithms,including Local S...Distributed stochastic gradient descent and its variants have been widely adopted in the training of machine learning models,which apply multiple workers in parallel.Among them,local-based algorithms,including Local SGD and FedAvg,have gained much attention due to their superior properties,such as low communication cost and privacypreserving.Nevertheless,when the data distribution on workers is non-identical,local-based algorithms would encounter a significant degradation in the convergence rate.In this paper,we propose Variance Reduced Local SGD(VRL-SGD)to deal with the heterogeneous data.Without extra communication cost,VRL-SGD can reduce the gradient variance among workers caused by the heterogeneous data,and thus it prevents local-based algorithms from slow convergence rate.Moreover,we present VRL-SGD-W with an effectivewarm-up mechanism for the scenarios,where the data among workers are quite diverse.Benefiting from eliminating the impact of such heterogeneous data,we theoretically prove that VRL-SGD achieves a linear iteration speedup with lower communication complexity even if workers access non-identical datasets.We conduct experiments on three machine learning tasks.The experimental results demonstrate that VRL-SGD performs impressively better than Local SGD for the heterogeneous data and VRL-SGD-W is much robust under high data variance among workers.展开更多
Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurr...Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurrence of non-independent and identically distributed(non-IID)and long-tailed distribution in FL is one challenge that substantially degrades aggregate performance.In this paper,we present a corresponding solution called federated dual-decoupling via model and logit calibration(FedDDC)for non-IID and long-tailed distributions.The model is characterized by three aspects.First,we decouple the global model into the feature extractor and the classifier to fine-tune the components affected by the joint problem.For the biased feature extractor,we propose a client confidence re-weighting scheme to assist calibration,which assigns optimal weights to each client.For the biased classifier,we apply the classifier re-balancing method for fine-tuning.Then,we calibrate and integrate the client confidence re-weighted logits with the re-balanced logits to obtain the unbiased logits.Finally,we use decoupled knowledge distillation for the first time in the joint problem to enhance the accuracy of the global model by extracting the knowledge of the unbiased model.Numerous experiments demonstrate that on non-IID and long-tailed data in FL,our approach outperforms state-of-the-art methods.展开更多
针对非独立同分布(Non-IID)场景下,联邦学习系统面临收敛缓慢和模型准确率降低等挑战,提出联邦学习中基于条件生成对抗网络的数据增强方案(FDA-GAN)。首先,设计一种类别选择的条件生成器为每个类别添加独立的网络模块,并将标签作为条件...针对非独立同分布(Non-IID)场景下,联邦学习系统面临收敛缓慢和模型准确率降低等挑战,提出联邦学习中基于条件生成对抗网络的数据增强方案(FDA-GAN)。首先,设计一种类别选择的条件生成器为每个类别添加独立的网络模块,并将标签作为条件信息,以更精确地提取各类别的特定特征;其次,提出一种覆盖类别的客户端选择策略来基于客户端的综合奖励,选择包含尽可能多类别的客户端集合参与训练,确保生成对抗网络(GAN)能学习到完整的类别分布;最后,利用生成样本扩充客户端的本地数据集,以优化本地数据的特征构成,减小客户端之间的偏差。实验结果表明,FDA-GAN在狄利克雷数据划分下,相较于CAP-GAN(Collaborated gAme Parallel learning based on GAN)的MNIST Score(MNIST inception Score)和Mode Score指标上分别提升了2.67和1.08,在FID(Fréchet Inception Distance)和MMD(Maximum Mean Discrepancy)指标上分别降低了55.12和2.56;在不同的Non-IID场景下,FedAvg(Federated Averaging)和FedProx(Federated Proximal)算法在结合FDA-GAN后,在50轮通信轮次内达到收敛,并且准确率提升了至少30.36个百分点。可见,FDA-GAN可以提高生成样本的质量与多样性,而且与基线算法结合后可以大幅提高联邦模型的准确率和收敛速度。展开更多
基金supported by the National Natural Science Foundation of China(62462040)the Yunnan Fundamental Research Projects(202501AT070345)the Major Science and Technology Projects in Yunnan Province(202202AD080013).
文摘Federated learning often experiences slow and unstable convergence due to edge-side data heterogeneity.This problem becomes more severe when edge participation rate is low,as the information collected from different edge devices varies significantly.As a result,communication overhead increases,which further slows down the convergence process.To address this challenge,we propose a simple yet effective federated learning framework that improves consistency among edge devices.The core idea is clusters the lookahead gradients collected from edge devices on the cloud server to obtain personalized momentum for steering local updates.In parallel,a global momentum is applied during model aggregation,enabling faster convergence while preserving personalization.This strategy enables efficient propagation of the estimated global update direction to all participating edge devices and maintains alignment in local training,without introducing extra memory or communication overhead.We conduct extensive experiments on benchmark datasets such as Cifar100 and Tiny-ImageNet.The results confirm the effectiveness of our framework.On CIFAR-100,our method reaches 55%accuracy with 37 fewer rounds and achieves a competitive final accuracy of 65.46%.Even under extreme non-IID scenarios,it delivers significant improvements in both accuracy and communication efficiency.The implementation is publicly available at https://github.com/sjmp525/CollaborativeComputing/tree/FedCCM(accessed on 20 October 2025).
文摘联邦学习因具有隐私保护的天然特性,已经逐渐成为一个被广泛认可的分布式机器学习框架。但由于参与方数据分布的差异性,特别是呈现非独立同分布(Non-Independent and Identically Distributed,Non-IID)时,其面临着泛化性能不足、收敛性能下降、数据倾斜等严峻挑战。用预训练基础模型缓解Non-IID问题作为一种新颖的方法,演变出了各种各样的解决方案。对此,从预训练基础模型的角度,对现有工作进行了综述。首先介绍了基础模型方法,对典型的基础模型编码结构进行对比分析。其次从修改输入、基础模型部分结构再训练,以及参数高效微调3个角度,提出了一种新的分类方法。最后探讨了该类工作的核心难题和未来研究方向。
基金This work was supported by the National Key R&D Program of China under Grant 2023YFB2703802the Hunan Province Innovation and Entrepreneurship Training Program for College Students S202311528073.
文摘Sharing data while protecting privacy in the industrial Internet is a significant challenge.Traditional machine learning methods require a combination of all data for training;however,this approach can be limited by data availability and privacy concerns.Federated learning(FL)has gained considerable attention because it allows for decentralized training on multiple local datasets.However,the training data collected by data providers are often non-independent and identically distributed(non-IID),resulting in poor FL performance.This paper proposes a privacy-preserving approach for sharing non-IID data in the industrial Internet using an FL approach based on blockchain technology.To overcome the problem of non-IID data leading to poor training accuracy,we propose dynamically updating the local model based on the divergence of the global and local models.This approach can significantly improve the accuracy of FL training when there is relatively large dispersion.In addition,we design a dynamic gradient clipping algorithm to alleviate the influence of noise on the model accuracy to reduce potential privacy leakage caused by sharing model parameters.Finally,we evaluate the performance of the proposed scheme using commonly used open-source image datasets.The simulation results demonstrate that our method can significantly enhance the accuracy while protecting privacy and maintaining efficiency,thereby providing a new solution to data-sharing and privacy-protection challenges in the industrial Internet.
文摘Federated learning(FL) is a machine learning paradigm for data silos and privacy protection,which aims to organize multiple clients for training global machine learning models without exposing data to all parties.However,when dealing with non-independently identically distributed(non-ⅡD) client data,FL cannot obtain more satisfactory results than centrally trained machine learning and even fails to match the accuracy of the local model obtained by client training alone.To analyze and address the above issues,we survey the state-of-theart methods in the literature related to FL on non-ⅡD data.On this basis,a motivation-based taxonomy,which classifies these methods into two categories,including heterogeneity reducing strategies and adaptability enhancing strategies,is proposed.Moreover,the core ideas and main challenges of these methods are analyzed.Finally,we envision several promising research directions that have not been thoroughly studied,in hope of promoting research in related fields to a certain extent.
文摘In non-independent and identically distributed(non-IID)data environments,model performance often degrades significantly.To address this issue,two improvement methods are proposed:FedReg and FedReg^(*).FedReg is a method based on hybrid regularization aimed at enhancing federated learning in non-IID scenarios.It introduces hybrid regularization to replace traditional L2 regularization,combining the advantages of L1 and L2 regularization to enable feature selection while preventing overfitting.This method better adapts to the diverse data distributions of different clients,improving the overall model performance.FedReg^(*)combines hybrid regularization with weighted model aggregation.In addition to the benefits of hybrid regularization,FedReg^(*)applies a weighted averaging method in the model aggregation process,calculating weights based on the cosine similarity between each client gradient and the global gradient to more reasonably distribute client contributions.By considering variations in data quality and quantity among clients,FedReg^(*)highlights the importance of key clients and enhances the model’s generalization performance.These improvement methods enhance model accuracy and communication efficiency.
文摘With the increasing complexity of vehicular networks and the proliferation of connected vehicles,Federated Learning(FL)has emerged as a critical framework for decentralized model training while preserving data privacy.However,efficient client selection and adaptive weight allocation in heterogeneous and non-IID environments remain challenging.To address these issues,we propose Federated Learning with Client Selection and Adaptive Weighting(FedCW),a novel algorithm that leverages adaptive client selection and dynamic weight allocation for optimizing model convergence in real-time vehicular networks.FedCW selects clients based on their Euclidean distance from the global model and dynamically adjusts aggregation weights to optimize both data diversity and model convergence.Experimental results show that FedCW significantly outperforms existing FL algorithms such as FedAvg,FedProx,and SCAFFOLD,particularly in non-IID settings,achieving faster convergence,higher accuracy,and reduced communication overhead.These findings demonstrate that FedCW provides an effective solution for enhancing the performance of FL in heterogeneous,edge-based computing environments.
基金This research was partially supported by grants from the National Key Research and Development Program of China(No.2018YFC0832101)the National Natural Science Foundation of China(Grant Nos.U20A20229 and 61922073).
文摘Distributed stochastic gradient descent and its variants have been widely adopted in the training of machine learning models,which apply multiple workers in parallel.Among them,local-based algorithms,including Local SGD and FedAvg,have gained much attention due to their superior properties,such as low communication cost and privacypreserving.Nevertheless,when the data distribution on workers is non-identical,local-based algorithms would encounter a significant degradation in the convergence rate.In this paper,we propose Variance Reduced Local SGD(VRL-SGD)to deal with the heterogeneous data.Without extra communication cost,VRL-SGD can reduce the gradient variance among workers caused by the heterogeneous data,and thus it prevents local-based algorithms from slow convergence rate.Moreover,we present VRL-SGD-W with an effectivewarm-up mechanism for the scenarios,where the data among workers are quite diverse.Benefiting from eliminating the impact of such heterogeneous data,we theoretically prove that VRL-SGD achieves a linear iteration speedup with lower communication complexity even if workers access non-identical datasets.We conduct experiments on three machine learning tasks.The experimental results demonstrate that VRL-SGD performs impressively better than Local SGD for the heterogeneous data and VRL-SGD-W is much robust under high data variance among workers.
基金supported by the National Natural Science Foundation of China(No.61702321)。
文摘Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurrence of non-independent and identically distributed(non-IID)and long-tailed distribution in FL is one challenge that substantially degrades aggregate performance.In this paper,we present a corresponding solution called federated dual-decoupling via model and logit calibration(FedDDC)for non-IID and long-tailed distributions.The model is characterized by three aspects.First,we decouple the global model into the feature extractor and the classifier to fine-tune the components affected by the joint problem.For the biased feature extractor,we propose a client confidence re-weighting scheme to assist calibration,which assigns optimal weights to each client.For the biased classifier,we apply the classifier re-balancing method for fine-tuning.Then,we calibrate and integrate the client confidence re-weighted logits with the re-balanced logits to obtain the unbiased logits.Finally,we use decoupled knowledge distillation for the first time in the joint problem to enhance the accuracy of the global model by extracting the knowledge of the unbiased model.Numerous experiments demonstrate that on non-IID and long-tailed data in FL,our approach outperforms state-of-the-art methods.
文摘针对非独立同分布(Non-IID)场景下,联邦学习系统面临收敛缓慢和模型准确率降低等挑战,提出联邦学习中基于条件生成对抗网络的数据增强方案(FDA-GAN)。首先,设计一种类别选择的条件生成器为每个类别添加独立的网络模块,并将标签作为条件信息,以更精确地提取各类别的特定特征;其次,提出一种覆盖类别的客户端选择策略来基于客户端的综合奖励,选择包含尽可能多类别的客户端集合参与训练,确保生成对抗网络(GAN)能学习到完整的类别分布;最后,利用生成样本扩充客户端的本地数据集,以优化本地数据的特征构成,减小客户端之间的偏差。实验结果表明,FDA-GAN在狄利克雷数据划分下,相较于CAP-GAN(Collaborated gAme Parallel learning based on GAN)的MNIST Score(MNIST inception Score)和Mode Score指标上分别提升了2.67和1.08,在FID(Fréchet Inception Distance)和MMD(Maximum Mean Discrepancy)指标上分别降低了55.12和2.56;在不同的Non-IID场景下,FedAvg(Federated Averaging)和FedProx(Federated Proximal)算法在结合FDA-GAN后,在50轮通信轮次内达到收敛,并且准确率提升了至少30.36个百分点。可见,FDA-GAN可以提高生成样本的质量与多样性,而且与基线算法结合后可以大幅提高联邦模型的准确率和收敛速度。
