In the current work,we explored a new knowledge amalgama-tion problem,termed Federated Selective Aggregation for on-device knowledge amalgamation(FedSA).FedSA aims to train an on-device student model for a new task wi...In the current work,we explored a new knowledge amalgama-tion problem,termed Federated Selective Aggregation for on-device knowledge amalgamation(FedSA).FedSA aims to train an on-device student model for a new task with the help of several decentralized teachers whose pre-training tasks and data are different and agnos-tic.The motivation to investigate such a problem setup stems from a recent dilemma of model sharing.Due to privacy,security or in-tellectual property issues,the pre-trained models are,however,not able to be shared,and the resources of devices are usually limited.The proposed FedSA offers a solution to this dilemma and makes it one step further,again,the method can be employed on low-power and resource-limited devices.To this end,a dedicated strategy was proposed to handle the knowledge amalgamation.Specifically,the student-training process in the current work was driven by a novel saliency-based approach which adaptively selects teachers as the par-ticipants and integrated their representative capabilities into the stu-dent.To evaluate the effectiveness of FedSA,experiments on both single-task and multi-task settings were conducted.The experimental results demonstrate that FedSA could effectively amalgamate knowl-edge from decentralized models and achieve competitive performance to centralized baselines.展开更多
基金supported by National Natural Science Foundation of China (61976186,U20B2066)the Fundamental Research Funds for the Central Universities (2021FZZX001-23,226-2023-00048).
文摘In the current work,we explored a new knowledge amalgama-tion problem,termed Federated Selective Aggregation for on-device knowledge amalgamation(FedSA).FedSA aims to train an on-device student model for a new task with the help of several decentralized teachers whose pre-training tasks and data are different and agnos-tic.The motivation to investigate such a problem setup stems from a recent dilemma of model sharing.Due to privacy,security or in-tellectual property issues,the pre-trained models are,however,not able to be shared,and the resources of devices are usually limited.The proposed FedSA offers a solution to this dilemma and makes it one step further,again,the method can be employed on low-power and resource-limited devices.To this end,a dedicated strategy was proposed to handle the knowledge amalgamation.Specifically,the student-training process in the current work was driven by a novel saliency-based approach which adaptively selects teachers as the par-ticipants and integrated their representative capabilities into the stu-dent.To evaluate the effectiveness of FedSA,experiments on both single-task and multi-task settings were conducted.The experimental results demonstrate that FedSA could effectively amalgamate knowl-edge from decentralized models and achieve competitive performance to centralized baselines.
