摘要
分布式机器学习因其优越的并行能力成为人工智能领域复杂模型训练的常用技术。然而,GPU升级换代非常快,异构集群环境下的分布式机器学习成为数据中心、研究机构面临的新常态。异构节点之间训练速度的差异使得现有并行方法难以平衡同步等待和陈旧梯度的影响,从而显著降低模型整体训练效率。针对该问题,提出了一种基于节点状态的动态自适应并行方法(dynamic adaptive synchronous parallel,DASP),利用参数服务器动态管理节点训练时的状态信息并对节点的并行状态进行划分,通过节点状态信息自适应调整每个节点的并行状态,以减少快速节点对全局模型参数的同步等待时间与陈旧梯度的产生,从而加快收敛效率。在公开数据集上的实验结果表明,DASP比主流方法收敛时间减少了16.9%~82.1%,并且训练过程更加稳定。
Distributed machine learning has emerged as a common technique for training complex artificial intelligence models due to its excellent parallelism capability.However,GPU upgrades are exceedingly fast,and distributed ma-chine learning in a heterogeneous cluster environment is increasingly being adopted by data centers and research institu-tions.The difference in training speed between heterogeneous nodes makes it difficult for existing parallel strategies to balance the effects of synchronized waits and stale gradients,considerably reducing the model’s overall training effi-ciency.To address this problem,a node state-based dynamic adaptive parallel strategy,namely,dynamic adaptive syn-chronous parallel(DASP),is proposed using a parameter server to dynamically manage the state information of nodes during training and to divide the parallel states of nodes.The parallel state of each node is adaptively adjusted by the state information of the node to reduce the synchronization waiting time of fast nodes for global model parameters and the generation of stale gradients,speeding up the convergence efficiency.Experimental results on publicly available datasets show that DASP not only reduces the convergence time by 16.9%~82.1%compared to mainstream strategies but also makes the training process more stable.
作者
马翔
申国伟
郭春
崔允贺
陈意
MA Xiang;SHEN Guowei;GUO Chun;CUI Yunhe;CHEN Yi(College of Computer Science and Technology,Guizhou University,Guiyang 550025,China)
出处
《智能系统学报》
CSCD
北大核心
2023年第5期1099-1107,共9页
CAAI Transactions on Intelligent Systems
基金
国家自然科学基金项目(62062022)。
关键词
异构集群
机器学习
数据并行
分布式训练
参数服务器
落后者
陈旧梯度
大规模深度学习
heterogeneous clusters
machine learning
data parallel
distributed training
parameter servers
stragglers
stale gradient
large-scale deep learning©《智能系统学报》编辑部版权所有
