Supporting real-time and privacy-preserving learning at the edge is emerging as a critical trend,bringing forth substantial challenges for deep learning(DL)training in the context of limited GPU(graphic processing uni...Supporting real-time and privacy-preserving learning at the edge is emerging as a critical trend,bringing forth substantial challenges for deep learning(DL)training in the context of limited GPU(graphic processing unit)memory.Recent work has sought to address the limitations by swapping tensors between GPU memory and CPU memory.Unfortunately,their tensor-based memory management encounters additional overhead since the swapped tensors do not align with the actual memory demands,resulting in decreased throughput.This paper introduces a vTensor-based memory management approach designed to mitigate memory swapping overhead.Virtualized tensors,dubbed vTensors,are used to finely align memory swapping amounts with real-time memory demands.Firstly,we introduce an abstraction layer that virtualizes coarse-grained tensors to multiple finer-grained vTensors.Secondly,we propose the Layered Graph Model(LGM)for analyzing vTensor mappings,which produces a memory swapping plan leveraged in the subsequent DL training iterations.Evaluations conducted on typical edge deep learning models illustrate that our approach surpasses prior work with a 15.60%increase in DL training throughput.展开更多
基金supported by the National Key Research and Development Program of China under Grant No.2023YFB3308702the National Natural Science Foundation of China under Grant No.62302489the Strategy Priority Research Program of the Chinese Academy of Sciences under Grant No.XDA0360301.
文摘Supporting real-time and privacy-preserving learning at the edge is emerging as a critical trend,bringing forth substantial challenges for deep learning(DL)training in the context of limited GPU(graphic processing unit)memory.Recent work has sought to address the limitations by swapping tensors between GPU memory and CPU memory.Unfortunately,their tensor-based memory management encounters additional overhead since the swapped tensors do not align with the actual memory demands,resulting in decreased throughput.This paper introduces a vTensor-based memory management approach designed to mitigate memory swapping overhead.Virtualized tensors,dubbed vTensors,are used to finely align memory swapping amounts with real-time memory demands.Firstly,we introduce an abstraction layer that virtualizes coarse-grained tensors to multiple finer-grained vTensors.Secondly,we propose the Layered Graph Model(LGM)for analyzing vTensor mappings,which produces a memory swapping plan leveraged in the subsequent DL training iterations.Evaluations conducted on typical edge deep learning models illustrate that our approach surpasses prior work with a 15.60%increase in DL training throughput.
