Neural network pruning is a popular approach to reducing the computational complexity of deep neural networks.In recent years,as growing evidence shows that conventional network pruning methods employ inappropriate pr...Neural network pruning is a popular approach to reducing the computational complexity of deep neural networks.In recent years,as growing evidence shows that conventional network pruning methods employ inappropriate proxy metrics,and as new types of hardware become increasingly available,hardware-aware network pruning that incorporates hardware characteristics in the loop of network pruning has gained growing attention,Both network accuracy and hardware efficiency(latency,memory consumption,etc.)are critical objectives to the success of network pruning,but the conflict between the multiple objectives makes it impossible to find a single optimal solution.Previous studies mostly convert the hardware-aware network pruning to optimization problems with a single objective.In this paper,we propose to solve the hardware-aware network pruning problem with Multi-Objective Evolutionary Algorithms(MOEAs).Specifically,we formulate the problem as a multi-objective optimization problem,and propose a novel memetic MOEA,namely HAMP,that combines an efficient portfoliobased selection and a surrogate-assisted local search,to solve it.Empirical studies demonstrate the potential of MOEAs in providing simultaneously a set of alternative solutions and the superiority of HAMP compared to the state-of-the-art hardware-aware network pruning method.展开更多
Filter pruning effectively compresses the neural network by reducing both its parameters and computational cost.Existing pruning methods typically rely on pre-designed pruning criteria to measure filter importance and...Filter pruning effectively compresses the neural network by reducing both its parameters and computational cost.Existing pruning methods typically rely on pre-designed pruning criteria to measure filter importance and remove those deemed unimportant.However,different layers of the neural network exhibit varying filter distributions,making it inappropriate to implement the same pruning criterion for all layers.Additionally,some approaches apply different criteria from the set of pre-defined pruning rules for different layers,but the limited space leads to the difficulty of covering all layers.If criteria for all layers are manually designed,it is costly and difficult to generalize to other networks.To solve this problem,we present a novel neural network pruning method based on the Criterion Learner and Attention Distillation(CLAD).Specifically,CLAD develops a differentiable criterion learner,which is integrated into each layer of the network.The learner can automatically learn the appropriate pruning criterion according to the filter parameters of each layer,thus the requirement of manual design is eliminated.Furthermore,the criterion learner is trained end-to-end by the gradient optimization algorithm to achieve efficient pruning.In addition,attention distillation,which fully utilizes the knowledge of unpruned networks to guide the optimization of the learner and improve the pruned network performance,is introduced in the process of learner optimization.Experiments conducted on various datasets and networks demonstrate the effectiveness of the proposed method.Notably,CLAD reduces the FLOPs of Res Net-110 by about 53%on the CIFAR-10 dataset,while simultaneously improves the network's accuracy by 0.05%.Moreover,it reduces the FLOPs of Res Net-50 by about 46%on the Image Net-1K dataset,and maintains a top-1 accuracy of 75.45%.展开更多
Graph neural networks (GNNs) have gained increasing popularity, while usually suffering from unaffordable computations for real-world large-scale applications. Hence, pruning GNNs is of great need but largely unexplor...Graph neural networks (GNNs) have gained increasing popularity, while usually suffering from unaffordable computations for real-world large-scale applications. Hence, pruning GNNs is of great need but largely unexplored. The recent work Unified GNN Sparsification (UGS) studies lottery ticket learning for GNNs, aiming to find a subset of model parameters and graph structures that can best maintain the GNN performance. However, it is tailed for the transductive setting, failing to generalize to unseen graphs, which are common in inductive tasks like graph classification. In this work, we propose a simple and effective learning paradigm, Inductive Co-Pruning of GNNs (ICPG), to endow graph lottery tickets with inductive pruning capacity. To prune the input graphs, we design a predictive model to generate importance scores for each edge based on the input. To prune the model parameters, it views the weight’s magnitude as their importance scores. Then we design an iterative co-pruning strategy to trim the graph edges and GNN weights based on their importance scores. Although it might be strikingly simple, ICPG surpasses the existing pruning method and can be universally applicable in both inductive and transductive learning settings. On 10 graph-classification and two node-classification benchmarks, ICPG achieves the same performance level with 14.26%–43.12% sparsity for graphs and 48.80%–91.41% sparsity for the GNN model.展开更多
基金the National Natural Science Foundation of China(62106098)the Stable Support Plan Program of Shenzhen Natural Science Fund(20200925154942002)the M0E University Scientific-Technological Innovation Plan Program.
文摘Neural network pruning is a popular approach to reducing the computational complexity of deep neural networks.In recent years,as growing evidence shows that conventional network pruning methods employ inappropriate proxy metrics,and as new types of hardware become increasingly available,hardware-aware network pruning that incorporates hardware characteristics in the loop of network pruning has gained growing attention,Both network accuracy and hardware efficiency(latency,memory consumption,etc.)are critical objectives to the success of network pruning,but the conflict between the multiple objectives makes it impossible to find a single optimal solution.Previous studies mostly convert the hardware-aware network pruning to optimization problems with a single objective.In this paper,we propose to solve the hardware-aware network pruning problem with Multi-Objective Evolutionary Algorithms(MOEAs).Specifically,we formulate the problem as a multi-objective optimization problem,and propose a novel memetic MOEA,namely HAMP,that combines an efficient portfoliobased selection and a surrogate-assisted local search,to solve it.Empirical studies demonstrate the potential of MOEAs in providing simultaneously a set of alternative solutions and the superiority of HAMP compared to the state-of-the-art hardware-aware network pruning method.
基金supported in part by the National Natural Science Foundation of China under grants 62073085,61973330 and 62350055in part by the Shenzhen Science and Technology Program,China under grant JCYJ20230807093513027in part by the Fundamental Research Funds for the Central Universities,China under grant 1243300008。
文摘Filter pruning effectively compresses the neural network by reducing both its parameters and computational cost.Existing pruning methods typically rely on pre-designed pruning criteria to measure filter importance and remove those deemed unimportant.However,different layers of the neural network exhibit varying filter distributions,making it inappropriate to implement the same pruning criterion for all layers.Additionally,some approaches apply different criteria from the set of pre-defined pruning rules for different layers,but the limited space leads to the difficulty of covering all layers.If criteria for all layers are manually designed,it is costly and difficult to generalize to other networks.To solve this problem,we present a novel neural network pruning method based on the Criterion Learner and Attention Distillation(CLAD).Specifically,CLAD develops a differentiable criterion learner,which is integrated into each layer of the network.The learner can automatically learn the appropriate pruning criterion according to the filter parameters of each layer,thus the requirement of manual design is eliminated.Furthermore,the criterion learner is trained end-to-end by the gradient optimization algorithm to achieve efficient pruning.In addition,attention distillation,which fully utilizes the knowledge of unpruned networks to guide the optimization of the learner and improve the pruned network performance,is introduced in the process of learner optimization.Experiments conducted on various datasets and networks demonstrate the effectiveness of the proposed method.Notably,CLAD reduces the FLOPs of Res Net-110 by about 53%on the CIFAR-10 dataset,while simultaneously improves the network's accuracy by 0.05%.Moreover,it reduces the FLOPs of Res Net-50 by about 46%on the Image Net-1K dataset,and maintains a top-1 accuracy of 75.45%.
基金supported by the National Key Research and Development Program of China under Grant No.2020YFB1406703the National Natural Science Foundation of China under Grant No.9227010114.
文摘Graph neural networks (GNNs) have gained increasing popularity, while usually suffering from unaffordable computations for real-world large-scale applications. Hence, pruning GNNs is of great need but largely unexplored. The recent work Unified GNN Sparsification (UGS) studies lottery ticket learning for GNNs, aiming to find a subset of model parameters and graph structures that can best maintain the GNN performance. However, it is tailed for the transductive setting, failing to generalize to unseen graphs, which are common in inductive tasks like graph classification. In this work, we propose a simple and effective learning paradigm, Inductive Co-Pruning of GNNs (ICPG), to endow graph lottery tickets with inductive pruning capacity. To prune the input graphs, we design a predictive model to generate importance scores for each edge based on the input. To prune the model parameters, it views the weight’s magnitude as their importance scores. Then we design an iterative co-pruning strategy to trim the graph edges and GNN weights based on their importance scores. Although it might be strikingly simple, ICPG surpasses the existing pruning method and can be universally applicable in both inductive and transductive learning settings. On 10 graph-classification and two node-classification benchmarks, ICPG achieves the same performance level with 14.26%–43.12% sparsity for graphs and 48.80%–91.41% sparsity for the GNN model.
