Industrial big data was usually multi-source, heterogeneous, and deeply intertwined. It had a wide range of data sources, high data dimensions, and strong data correlation. In order to effectively analyze and process ...Industrial big data was usually multi-source, heterogeneous, and deeply intertwined. It had a wide range of data sources, high data dimensions, and strong data correlation. In order to effectively analyze and process streaming industrial big data generated by edge computing, it was very important to provide an effective real-time incremental data method. However, in the process of incremental processing, industrial big data incremental computing faced the challenges of dimensional disaster, repeated calculations, and the explosion of intermediate results. Therefore, in order to solve the above problems effectively, a QR-based tensor-train(TT) decomposition(TTD) method and a QR-based incremental TTD(QRITTD) method were proposed. This algorithm combined the incremental QR-based decomposition algorithm with an approximate singular value decomposition(SVD) algorithm and had good scalability. In addition, the computational complexity, space complexity, and approximation error analysis were analyzed in detail. The effectiveness of the three algorithms of QRITTD, non-incremental TTD(NITTD), and TT rank-1(TTr1) SVD(TTr1 SVD)were verified by comparison. Experimental results show that the SVD QRITTD method has better performance under the premise of ensuring the same tensor size.展开更多
随着大语言模型(large language models,LLMs)(以下简称“大模型”)参数规模的持续增长,微调百亿级参数大模型对计算和存储资源提出了极高要求。传统分布式训练方案通常依赖大量高端GPU和高速互联网络,训练成本极为昂贵。现有单GPU训练...随着大语言模型(large language models,LLMs)(以下简称“大模型”)参数规模的持续增长,微调百亿级参数大模型对计算和存储资源提出了极高要求。传统分布式训练方案通常依赖大量高端GPU和高速互联网络,训练成本极为昂贵。现有单GPU训练方案虽通过张量卸载缓解显存压力,但仍然面临I/O传输效率低和设备利用率不足等问题。传统内核态I/O操作在大规模张量迁移中引入频繁的系统调用和上下文切换,成为制约性能的关键瓶颈;同时,优化器计算无法充分发挥多核CPU的并行能力,难以实现与GPU计算的有效重叠,进一步限制了系统性能。针对上述问题,提出了一种面向大模型训练的异构内存卸载与I/O优化方案HiTrain。首先构建了基于存储性能开发工具包(storage performance development kit,SPDK)的高性能张量存储模块,通过在用户态管理张量数据,避免了内核I/O栈开销,从而提高张量卸载的并发性与吞吐率;其次,设计并实现了基于异步优化器的存储-计算流水线调度模块,通过对优化器的执行进行优化重排来减少GPU等待时间,提高整体训练效率。实验结果表明,在配备单张GPU和非易失性存储器快速固态硬盘(non-volatile memory express solid state drive,NVMe SSD)的服务器上,所提出的方案能够充分利用系统中的存算资源,使得模型训练过程中张量卸载与加载效率提升32.7%,整体训练吞吐提升至现有方案的1.49倍,为低成本大模型训练提供了切实可行的技术路径。展开更多
近期语言模型领域的进展显示,采用Transformer架构的大型预训练模型在自然语言处理应用中表现出优异的技术能力。然而,受限于GPU内存,训练大语言模型(large language models,LLMs)成为了一项挑战。张量并行方法要求单个GPU存储所有激活...近期语言模型领域的进展显示,采用Transformer架构的大型预训练模型在自然语言处理应用中表现出优异的技术能力。然而,受限于GPU内存,训练大语言模型(large language models,LLMs)成为了一项挑战。张量并行方法要求单个GPU存储所有激活值,难以突破内存瓶颈。为解决GPU内存对大语言模型训练的制约并提升训练效率,本文提出一种二维张量并行方法(2D tensor parallelism,TP2D)。2D张量并行将输入数据和参数矩阵分割并分配至4个GPU;采用分布式通信,进行GPU间数据的高速交互,实现真正的分布式并行训练。以GPT-2模型作为基准模型,测试了两种训练方法的软扩展(soft scaling)效率和训练效率。实验表明,当使用4块GPU时,2D张量并行的训练速度是张量并行的1.84倍,软扩展效率达到86%,并降低了内存占用。展开更多
近年来,基于张量补全的频谱制图得到了广泛研究.目前用于频谱制图的张量补全算法大多隐含地假设张量具有平衡特性,而对于非平衡张量,难以利用其低秩性估计完整的张量信息,导致补全算法性能受损.本文提出基于重叠Ket增强(Overlapping Ket...近年来,基于张量补全的频谱制图得到了广泛研究.目前用于频谱制图的张量补全算法大多隐含地假设张量具有平衡特性,而对于非平衡张量,难以利用其低秩性估计完整的张量信息,导致补全算法性能受损.本文提出基于重叠Ket增强(Overlapping Ket Augmentation,OKA)和张量列车(Tensor Train,TT)的非平衡频谱制图算法,以解决非平衡张量在应用传统张量补全算法时性能下降的问题.首先使用OKA将低阶高维张量表示为高阶低维张量,在无信息损耗的情况下解决非平衡张量无法利用其低秩性进行张量补全的问题;然后使用TT矩阵化得到较平衡的矩阵,在维度较平衡条件下提高补全算法的精确度;最后利用高阶低维张量的低秩性,使用并行矩阵分解或基于F范数的无奇异值分解(Singular Value Decomposition Free,SVDFree)算法完成张量补全.仿真结果表明,针对非平衡张量,所提方案与现有的张量补全算法相比,可以获得更精确的无线电地图,同时所提SVDFree算法具有更低的计算复杂度.展开更多
基金supported by the Science and Technology Project in Shaanxi Province of China (2019ZDLGY07-08)the Natural Science Foundation Research Program of Shaanxi Province, China。
文摘Industrial big data was usually multi-source, heterogeneous, and deeply intertwined. It had a wide range of data sources, high data dimensions, and strong data correlation. In order to effectively analyze and process streaming industrial big data generated by edge computing, it was very important to provide an effective real-time incremental data method. However, in the process of incremental processing, industrial big data incremental computing faced the challenges of dimensional disaster, repeated calculations, and the explosion of intermediate results. Therefore, in order to solve the above problems effectively, a QR-based tensor-train(TT) decomposition(TTD) method and a QR-based incremental TTD(QRITTD) method were proposed. This algorithm combined the incremental QR-based decomposition algorithm with an approximate singular value decomposition(SVD) algorithm and had good scalability. In addition, the computational complexity, space complexity, and approximation error analysis were analyzed in detail. The effectiveness of the three algorithms of QRITTD, non-incremental TTD(NITTD), and TT rank-1(TTr1) SVD(TTr1 SVD)were verified by comparison. Experimental results show that the SVD QRITTD method has better performance under the premise of ensuring the same tensor size.
文摘随着大语言模型(large language models,LLMs)(以下简称“大模型”)参数规模的持续增长,微调百亿级参数大模型对计算和存储资源提出了极高要求。传统分布式训练方案通常依赖大量高端GPU和高速互联网络,训练成本极为昂贵。现有单GPU训练方案虽通过张量卸载缓解显存压力,但仍然面临I/O传输效率低和设备利用率不足等问题。传统内核态I/O操作在大规模张量迁移中引入频繁的系统调用和上下文切换,成为制约性能的关键瓶颈;同时,优化器计算无法充分发挥多核CPU的并行能力,难以实现与GPU计算的有效重叠,进一步限制了系统性能。针对上述问题,提出了一种面向大模型训练的异构内存卸载与I/O优化方案HiTrain。首先构建了基于存储性能开发工具包(storage performance development kit,SPDK)的高性能张量存储模块,通过在用户态管理张量数据,避免了内核I/O栈开销,从而提高张量卸载的并发性与吞吐率;其次,设计并实现了基于异步优化器的存储-计算流水线调度模块,通过对优化器的执行进行优化重排来减少GPU等待时间,提高整体训练效率。实验结果表明,在配备单张GPU和非易失性存储器快速固态硬盘(non-volatile memory express solid state drive,NVMe SSD)的服务器上,所提出的方案能够充分利用系统中的存算资源,使得模型训练过程中张量卸载与加载效率提升32.7%,整体训练吞吐提升至现有方案的1.49倍,为低成本大模型训练提供了切实可行的技术路径。
文摘近年来,基于张量补全的频谱制图得到了广泛研究.目前用于频谱制图的张量补全算法大多隐含地假设张量具有平衡特性,而对于非平衡张量,难以利用其低秩性估计完整的张量信息,导致补全算法性能受损.本文提出基于重叠Ket增强(Overlapping Ket Augmentation,OKA)和张量列车(Tensor Train,TT)的非平衡频谱制图算法,以解决非平衡张量在应用传统张量补全算法时性能下降的问题.首先使用OKA将低阶高维张量表示为高阶低维张量,在无信息损耗的情况下解决非平衡张量无法利用其低秩性进行张量补全的问题;然后使用TT矩阵化得到较平衡的矩阵,在维度较平衡条件下提高补全算法的精确度;最后利用高阶低维张量的低秩性,使用并行矩阵分解或基于F范数的无奇异值分解(Singular Value Decomposition Free,SVDFree)算法完成张量补全.仿真结果表明,针对非平衡张量,所提方案与现有的张量补全算法相比,可以获得更精确的无线电地图,同时所提SVDFree算法具有更低的计算复杂度.
