Recent years have witnessed a spurt of progress in federated learning,which can coordinate multi-participation model training while protecting the data privacy of participants.However,low communication efficiency is a...Recent years have witnessed a spurt of progress in federated learning,which can coordinate multi-participation model training while protecting the data privacy of participants.However,low communication efficiency is a bottleneck when deploying federated learning to edge computing and IoT devices due to the need to transmit a huge number of parameters during co-training.In this paper,we verify that the outputs of the last hidden layer can record the characteristics of training data.Accordingly,we propose a communication-efficient strategy based on model split and representation aggregate.Specifically,we make the client upload the outputs of the last hidden layer instead of all model parameters when participating in the aggregation,and the server distributes gradients according to the global information to revise local models.Empirical evidence from experiments verifies that our method can complete training by uploading less than one-tenth of model parameters,while preserving the usability of the model.展开更多
Distributed computing is an important topic in the field of wireless communications and networking,and its high efficiency in handling large amounts of data is particularly noteworthy.Although distributed computing be...Distributed computing is an important topic in the field of wireless communications and networking,and its high efficiency in handling large amounts of data is particularly noteworthy.Although distributed computing benefits from its ability of processing data in parallel,the communication burden between different servers is incurred,thereby the computation process is detained.Recent researches have applied coding in distributed computing to reduce the communication burden,where repetitive computation is utilized to enable multicast opportunities so that the same coded information can be reused across different servers.To handle the computation tasks in practical heterogeneous systems,we propose a novel coding scheme to effectively mitigate the "straggling effect" in distributed computing.We assume that there are two types of servers in the system and the only difference between them is their computational capabilities,the servers with lower computational capabilities are called stragglers.Given any ratio of fast servers to slow servers and any gap of computational capabilities between them,we achieve approximately the same computation time for both fast and slow servers by assigning different amounts of computation tasks to them,thus reducing the overall computation time.Furthermore,we investigate the informationtheoretic lower bound of the inter-communication load and show that the lower bound is within a constant multiplicative gap to the upper bound achieved by our scheme.Various simulations also validate the effectiveness of the proposed scheme.展开更多
基金supported by Shenzhen Basic Research (General Project)under Grant No.JCYJ20190806142601687Shenzhen Stable Supporting Program (General Project) under Grant No.GXWD20201230155427003-20200821160539001+1 种基金Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies under Grant No.2022B1212010005Shenzhen Basic Research (Key Project) under Grant No.JCYJ20200109113405927。
文摘Recent years have witnessed a spurt of progress in federated learning,which can coordinate multi-participation model training while protecting the data privacy of participants.However,low communication efficiency is a bottleneck when deploying federated learning to edge computing and IoT devices due to the need to transmit a huge number of parameters during co-training.In this paper,we verify that the outputs of the last hidden layer can record the characteristics of training data.Accordingly,we propose a communication-efficient strategy based on model split and representation aggregate.Specifically,we make the client upload the outputs of the last hidden layer instead of all model parameters when participating in the aggregation,and the server distributes gradients according to the global information to revise local models.Empirical evidence from experiments verifies that our method can complete training by uploading less than one-tenth of model parameters,while preserving the usability of the model.
基金supported by NSF China(No.T2421002,62061146002,62020106005)。
文摘Distributed computing is an important topic in the field of wireless communications and networking,and its high efficiency in handling large amounts of data is particularly noteworthy.Although distributed computing benefits from its ability of processing data in parallel,the communication burden between different servers is incurred,thereby the computation process is detained.Recent researches have applied coding in distributed computing to reduce the communication burden,where repetitive computation is utilized to enable multicast opportunities so that the same coded information can be reused across different servers.To handle the computation tasks in practical heterogeneous systems,we propose a novel coding scheme to effectively mitigate the "straggling effect" in distributed computing.We assume that there are two types of servers in the system and the only difference between them is their computational capabilities,the servers with lower computational capabilities are called stragglers.Given any ratio of fast servers to slow servers and any gap of computational capabilities between them,we achieve approximately the same computation time for both fast and slow servers by assigning different amounts of computation tasks to them,thus reducing the overall computation time.Furthermore,we investigate the informationtheoretic lower bound of the inter-communication load and show that the lower bound is within a constant multiplicative gap to the upper bound achieved by our scheme.Various simulations also validate the effectiveness of the proposed scheme.
