Transformer models face significant computational challenges in private inference(PI).Existing optimization methods often rely on isolated techniques,neglecting joint structural and operational improvements.We propose...Transformer models face significant computational challenges in private inference(PI).Existing optimization methods often rely on isolated techniques,neglecting joint structural and operational improvements.We propose IG-3D,a unified framework that integrates structured compression and operator approximation through accurate importance assessment.Our approach first evaluates attention head importance using Integrated Gradients(IG),offering greater stability and theoretical soundness than gradient-based methods.We then apply a threedimensional optimization:(1)structurally pruning redundant attention heads;(2)replacing Softmax with adaptive polynomial approximation to avoid exponential computations;(3)implementing layer-wise GELU substitution to accommodate different layer characteristics.A joint thresholdmechanism coordinates compression across dimensions under accuracy constraints.Experimental results on the GLUE benchmark show that our method achieves an average 2.9×speedup in inference latency and a 50%reduction in communication cost,while controlling the accuracy loss within 2.3%,demonstrating significant synergistic effects and a superior accuracy-efficiency trade-off compared to single-technique optimization strategies.展开更多
The existing multi-objective wheel profile optimization methods mainly consist of three sub-modules:(1)wheel profile generation,(2)multi-body dynamics simulation,and(3)an optimization algorithm.For the first module,a ...The existing multi-objective wheel profile optimization methods mainly consist of three sub-modules:(1)wheel profile generation,(2)multi-body dynamics simulation,and(3)an optimization algorithm.For the first module,a comparably conservative rotary-scaling finetuning(RSFT)method,which introduces two design variables and an empirical formula,is proposed to fine-tune the traditional wheel profiles for improving their engineering applicability.For the second module,for the TRAXX locomotives serving on the Blankenburg–Rubeland line,an optimization function representing the relationship between the wheel profile and the wheel–rail wear number is established based on Kriging surrogate model(KSM).For the third module,a method combining the regression capability of KSM with the iterative computing power of particle swarm optimization(PSO)is proposed to quickly and reliably implement the task of optimizing wheel profiles.Finally,with the RSFT–KSM–PSO method,we propose two wear-resistant wheel profiles for the TRAXX locomotives serving on the Blankenburg–Rubeland line,namely S1002-S and S1002-M.The S1002-S profile minimizes the total wear number by 30%,while the S1002-M profile makes the wear distribution more uniform through a proper sacrifice of the tread wear number,and the total wear number is reduced by 21%.The quasi-static and hunting stability tests further demonstrate that the profile designed by the RSFT–KSM–PSO method is promising for practical engineering applications.展开更多
Automatic speech recognition(ASR)systems have emerged as indispensable tools across a wide spectrum of applications,ranging from transcription services to voice-activated assistants.To enhance the performance of these...Automatic speech recognition(ASR)systems have emerged as indispensable tools across a wide spectrum of applications,ranging from transcription services to voice-activated assistants.To enhance the performance of these systems,it is important to deploy efficient models capable of adapting to diverse deployment conditions.In recent years,on-demand pruning methods have obtained significant attention within the ASR domain due to their adaptability in various deployment scenarios.However,these methods often confront substantial trade-offs,particularly in terms of unstable accuracy when reducing the model size.To address challenges,this study introduces two crucial empirical findings.Firstly,it proposes the incorporation of an online distillation mechanism during on-demand pruning training,which holds the promise of maintaining more consistent accuracy levels.Secondly,it proposes the utilization of the Mogrifier long short-term memory(LSTM)language model(LM),an advanced iteration of the conventional LSTM LM,as an effective alternative for pruning targets within the ASR framework.Through rigorous experimentation on the ASR system,employing the Mogrifier LSTM LM and training it using the suggested joint on-demand pruning and online distillation method,this study provides compelling evidence.The results exhibit that the proposed methods significantly outperform a benchmark model trained solely with on-demand pruning methods.Impressively,the proposed strategic configuration successfully reduces the parameter count by approximately 39%,all the while minimizing trade-offs.展开更多
This paper develops a wheel profile fine-tuning system(WPFTS)that comprehensively considers the influence of wheel profile on wheel damage,vehicle stability,vehicle safety,and passenger comfort.WPFTS can recommend one...This paper develops a wheel profile fine-tuning system(WPFTS)that comprehensively considers the influence of wheel profile on wheel damage,vehicle stability,vehicle safety,and passenger comfort.WPFTS can recommend one or more optimized wheel profiles according to train operators’needs,e.g.,reducing wheel wear,mitigating the development of wheel out-of-roundness(OOR),improving the shape stability of the wheel profile.Specifically,WPFTS includes four modules:(I)a wheel profile generation module based on the rotary-scaling finetuning(RSFT)method;(II)a multi-objective generation module consisting of a rigid multi-body dynamics simulation(MBS)model,an analytical model,and a rigid–flexible MBS model,for generating 11 objectives related to wheel damage,vehicle stability,vehicle safety,and passenger comfort;(III)a weight assignment module consisting of an adaptive weight assignment strategy and a manual weight assignment strategy;and(IV)an optimization module based on radial basis function(RBF)and particle swarm optimization(PSO).Finally,three cases are introduced to show how WPTFS recommends a wheel profile according to train operators’needs.Among them,a wheel profile with high shape stability,a wheel profile for mitigating the development of wheel OOR,and a wheel profile considering hunting stability and derailment safety are developed,respectively.展开更多
Time series foundation models provide a universal solution for generating forecasts to support optimization problems in energy systems.Those foundation models are typically trained in a prediction-focused manner to ma...Time series foundation models provide a universal solution for generating forecasts to support optimization problems in energy systems.Those foundation models are typically trained in a prediction-focused manner to maximize forecast quality.In contrast,decision-focused learning directly improves the resulting value of the forecast in downstream optimization rather than merely maximizing forecasting quality.The practical integration of forecast values into forecasting models is challenging,particularly when addressing complex applications with diverse instances,such as buildings.This becomes even more complicated when instances possess specific characteristics that require instance-specific,tailored predictions to increase the forecast value.To tackle this challenge,we use decision-focused fine-tuning within time series foundation models to offer a scalable and efficient solution for decision-focused learning applied to the dispatchable feeder optimization problem.To obtain more robust predictions for scarce building data,we use Moirai as a state-of-the-art foundation model,which offers robust and generalized results with few-shot parameter-efficient fine-tuning.Comparing the decision-focused fine-tuned Moirai with a state-of-the-art classical prediction-focused fine-tuning Moirai,we observe an improvement of 9.45%in Average Daily Total Costs.展开更多
针对RRT(Rapidly-exploring Random Tree)算法在机器人路径规划过程存在采样点随机性高、算法效率低、路径规划时间长以及规划路径冗长等问题,文中提出一种结合人工势场法的双向RRT路径规划算法。将传统RRT算法中单向扩展方式改为由起...针对RRT(Rapidly-exploring Random Tree)算法在机器人路径规划过程存在采样点随机性高、算法效率低、路径规划时间长以及规划路径冗长等问题,文中提出一种结合人工势场法的双向RRT路径规划算法。将传统RRT算法中单向扩展方式改为由起点和终点同时进行扩展,在节点扩展时加入人工势场法进行引导,增加节点扩展的目的性。将固定步长改换为可变步长,使随机树可以更快地向目标点扩展。对生成路径进行剪枝处理,删除路径中的冗余节点,进一步缩短路径长度。利用MATLAB仿真平台在相同环境下对比所提改进算法与RRT-Connect算法、DRRT-Connect(Dynamic Rapidly-exploring Random Tree Connect)算法、GB(Goal-Biased)-RRT算法、A^(*)算法、PRM(Probabilistic Road Map)算法的路径规划效果。仿真结果表明,所提改进算法与其他改进算法相比最短路径缩短了7%,最短搜索时间降低了65%,提高了算法的规划效率。将所提算法应用于机器人,结果证明了其具有较强可行性。展开更多
A novel gravity assist space pruning(GASP)algorithm based on image tools is proposed for solving interplanetary trajectory optimization problem.Compared with traditional GASP algorithm,the concept of image is introduc...A novel gravity assist space pruning(GASP)algorithm based on image tools is proposed for solving interplanetary trajectory optimization problem.Compared with traditional GASP algorithm,the concept of image is introduced to avoid missing interesting solutions with appropriate number of function evaluations.Image tools allow us to evaluate the objective function in regions in place of points and provide an effective way to evaluate the forward and backward constraints for the multi-gravity assist trajectory optimization problem.Since the interesting solutions of the interplanetary trajectory optimization problem are often clustered in a small portion of the search space rather than being overall evenly distributed,the regionwise evaluations with image tools make the little large interval with the proper Lipschitzian tolerances sampling effective.The detailed steps of the proposed method are presented and two examples including Earth Venus Mars(EVM)transfer and Earth Venus Venus Earth Jupiter Saturn(EVVEJS)transfer are given.Finally,a comparison with solutions given by the literature demonstrates the effectiveness of the proposed method.展开更多
Aimed at the great computing complexity of optimal brain surgeon (OBS) process, a pruning algorithm with penalty OBS process is presented. Compared with sensitive and regularized methods, the penalty OBS algorithm not...Aimed at the great computing complexity of optimal brain surgeon (OBS) process, a pruning algorithm with penalty OBS process is presented. Compared with sensitive and regularized methods, the penalty OBS algorithm not only avoids time-consuming defect and low pruning efficiency in OBS process, but also keeps higher generalization and pruning accuracy than Levenberg-Marquardt method.展开更多
Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requir...Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.展开更多
针对快速扩展随机树(Rapidly-exploring Random Tree,RRT)算法及其衍生算法路径规划时间长且规划效率低的问题,提出RRT算法与人工势场法结合的PAAPF-RRT机械臂路径规划算法,旨在最短的时间、最小的迭代次数内,在静态环境中找到连接起始...针对快速扩展随机树(Rapidly-exploring Random Tree,RRT)算法及其衍生算法路径规划时间长且规划效率低的问题,提出RRT算法与人工势场法结合的PAAPF-RRT机械臂路径规划算法,旨在最短的时间、最小的迭代次数内,在静态环境中找到连接起始点与终点的最优路径。首先,引入基于地图障碍物分布评估策略和采样区域优化策略,根据地图的障碍物分布、数量调整算法的步长以及偏向概率。然后,伴随随机树的生长,更新随机点的采样区域,保证随机树向目标点生长。其次,将RRT算法与人工势场法结合,当随机树与障碍物发生碰撞时,使用人工势场法引导随机树节点生长避开障碍物,解决了RRT算法随机树生长到障碍物附近且朝目标点生长的方向被障碍物遮挡时随机树无法生长的问题。最后,利用节点修剪策略,把算法生成的初始路径中的冗余节点进行修剪,得到拐点更少、路径更简洁的优化路径。实验结果表明,PAAPF-RRT算法在路径规划时间上对于RRT算法、GB-RRT算法以及RRT*算法分别减少了93.64%、73.58%、93.28%,在迭代次数方面分别下降了91.40%、79.64%、90.58%,在路径长度方面只占其他3种算法的79.34%、86.21%、95.58%。展开更多
文摘Transformer models face significant computational challenges in private inference(PI).Existing optimization methods often rely on isolated techniques,neglecting joint structural and operational improvements.We propose IG-3D,a unified framework that integrates structured compression and operator approximation through accurate importance assessment.Our approach first evaluates attention head importance using Integrated Gradients(IG),offering greater stability and theoretical soundness than gradient-based methods.We then apply a threedimensional optimization:(1)structurally pruning redundant attention heads;(2)replacing Softmax with adaptive polynomial approximation to avoid exponential computations;(3)implementing layer-wise GELU substitution to accommodate different layer characteristics.A joint thresholdmechanism coordinates compression across dimensions under accuracy constraints.Experimental results on the GLUE benchmark show that our method achieves an average 2.9×speedup in inference latency and a 50%reduction in communication cost,while controlling the accuracy loss within 2.3%,demonstrating significant synergistic effects and a superior accuracy-efficiency trade-off compared to single-technique optimization strategies.
基金the Assets4Rail Project which is funded by the Shift2Rail Joint Undertaking under the EU’s H2020 program(Grant No.826250)the Open Research Fund of State Key Laboratory of Traction Power of Southwest Jiaotong University(Grant No.TPL2011)+1 种基金part of the experiment data concerning the railway line is supported by the DynoTRAIN Project,funded by European Commission(Grant No.234079)The first author is also supported by the China Scholarship Council(Grant No.201707000113).
文摘The existing multi-objective wheel profile optimization methods mainly consist of three sub-modules:(1)wheel profile generation,(2)multi-body dynamics simulation,and(3)an optimization algorithm.For the first module,a comparably conservative rotary-scaling finetuning(RSFT)method,which introduces two design variables and an empirical formula,is proposed to fine-tune the traditional wheel profiles for improving their engineering applicability.For the second module,for the TRAXX locomotives serving on the Blankenburg–Rubeland line,an optimization function representing the relationship between the wheel profile and the wheel–rail wear number is established based on Kriging surrogate model(KSM).For the third module,a method combining the regression capability of KSM with the iterative computing power of particle swarm optimization(PSO)is proposed to quickly and reliably implement the task of optimizing wheel profiles.Finally,with the RSFT–KSM–PSO method,we propose two wear-resistant wheel profiles for the TRAXX locomotives serving on the Blankenburg–Rubeland line,namely S1002-S and S1002-M.The S1002-S profile minimizes the total wear number by 30%,while the S1002-M profile makes the wear distribution more uniform through a proper sacrifice of the tread wear number,and the total wear number is reduced by 21%.The quasi-static and hunting stability tests further demonstrate that the profile designed by the RSFT–KSM–PSO method is promising for practical engineering applications.
基金supported by Institute of Information&communications Technology Planning&Evaluation(IITP)grant funded by the Korea government(MSIT)(No.2022-0-00377,Development of Intelligent Analysis and Classification Based Contents Class Categorization Technique to Prevent Imprudent Harmful Media Distribution).
文摘Automatic speech recognition(ASR)systems have emerged as indispensable tools across a wide spectrum of applications,ranging from transcription services to voice-activated assistants.To enhance the performance of these systems,it is important to deploy efficient models capable of adapting to diverse deployment conditions.In recent years,on-demand pruning methods have obtained significant attention within the ASR domain due to their adaptability in various deployment scenarios.However,these methods often confront substantial trade-offs,particularly in terms of unstable accuracy when reducing the model size.To address challenges,this study introduces two crucial empirical findings.Firstly,it proposes the incorporation of an online distillation mechanism during on-demand pruning training,which holds the promise of maintaining more consistent accuracy levels.Secondly,it proposes the utilization of the Mogrifier long short-term memory(LSTM)language model(LM),an advanced iteration of the conventional LSTM LM,as an effective alternative for pruning targets within the ASR framework.Through rigorous experimentation on the ASR system,employing the Mogrifier LSTM LM and training it using the suggested joint on-demand pruning and online distillation method,this study provides compelling evidence.The results exhibit that the proposed methods significantly outperform a benchmark model trained solely with on-demand pruning methods.Impressively,the proposed strategic configuration successfully reduces the parameter count by approximately 39%,all the while minimizing trade-offs.
基金This work was supported by China Scholarship Council(Grant No.201707000113).
文摘This paper develops a wheel profile fine-tuning system(WPFTS)that comprehensively considers the influence of wheel profile on wheel damage,vehicle stability,vehicle safety,and passenger comfort.WPFTS can recommend one or more optimized wheel profiles according to train operators’needs,e.g.,reducing wheel wear,mitigating the development of wheel out-of-roundness(OOR),improving the shape stability of the wheel profile.Specifically,WPFTS includes four modules:(I)a wheel profile generation module based on the rotary-scaling finetuning(RSFT)method;(II)a multi-objective generation module consisting of a rigid multi-body dynamics simulation(MBS)model,an analytical model,and a rigid–flexible MBS model,for generating 11 objectives related to wheel damage,vehicle stability,vehicle safety,and passenger comfort;(III)a weight assignment module consisting of an adaptive weight assignment strategy and a manual weight assignment strategy;and(IV)an optimization module based on radial basis function(RBF)and particle swarm optimization(PSO).Finally,three cases are introduced to show how WPTFS recommends a wheel profile according to train operators’needs.Among them,a wheel profile with high shape stability,a wheel profile for mitigating the development of wheel OOR,and a wheel profile considering hunting stability and derailment safety are developed,respectively.
基金funded by the Helmholtz Association’s Initiative and Networking Fund through Helmholtz AI,the Helmholtz Association under the Program“Energy System Design”the German Research Foundation(DFG)as part of the Research Training Group 2153“En-ergy Status Data:Informatics Methods for its Collection,Analysis and Exploitation”+1 种基金supported by the Helmholtz Association Initiative and Networking Fund on the HAICORE@KIT partitionsupport by the KIT-Publication Fund of the Karlsruhe Institute of Technology.
文摘Time series foundation models provide a universal solution for generating forecasts to support optimization problems in energy systems.Those foundation models are typically trained in a prediction-focused manner to maximize forecast quality.In contrast,decision-focused learning directly improves the resulting value of the forecast in downstream optimization rather than merely maximizing forecasting quality.The practical integration of forecast values into forecasting models is challenging,particularly when addressing complex applications with diverse instances,such as buildings.This becomes even more complicated when instances possess specific characteristics that require instance-specific,tailored predictions to increase the forecast value.To tackle this challenge,we use decision-focused fine-tuning within time series foundation models to offer a scalable and efficient solution for decision-focused learning applied to the dispatchable feeder optimization problem.To obtain more robust predictions for scarce building data,we use Moirai as a state-of-the-art foundation model,which offers robust and generalized results with few-shot parameter-efficient fine-tuning.Comparing the decision-focused fine-tuned Moirai with a state-of-the-art classical prediction-focused fine-tuning Moirai,we observe an improvement of 9.45%in Average Daily Total Costs.
文摘针对RRT(Rapidly-exploring Random Tree)算法在机器人路径规划过程存在采样点随机性高、算法效率低、路径规划时间长以及规划路径冗长等问题,文中提出一种结合人工势场法的双向RRT路径规划算法。将传统RRT算法中单向扩展方式改为由起点和终点同时进行扩展,在节点扩展时加入人工势场法进行引导,增加节点扩展的目的性。将固定步长改换为可变步长,使随机树可以更快地向目标点扩展。对生成路径进行剪枝处理,删除路径中的冗余节点,进一步缩短路径长度。利用MATLAB仿真平台在相同环境下对比所提改进算法与RRT-Connect算法、DRRT-Connect(Dynamic Rapidly-exploring Random Tree Connect)算法、GB(Goal-Biased)-RRT算法、A^(*)算法、PRM(Probabilistic Road Map)算法的路径规划效果。仿真结果表明,所提改进算法与其他改进算法相比最短路径缩短了7%,最短搜索时间降低了65%,提高了算法的规划效率。将所提算法应用于机器人,结果证明了其具有较强可行性。
基金supported by the National High Technology Research and Development Program (863)of China (2012AA121602)the National Natural Science Foundation of China(11078001)+1 种基金the Specialized Research Fund for the Doctoral Program of Higher Education of China (20133218120037)the Fundamental Research Funds for the Central Universities under Grant(NS2014091)
文摘A novel gravity assist space pruning(GASP)algorithm based on image tools is proposed for solving interplanetary trajectory optimization problem.Compared with traditional GASP algorithm,the concept of image is introduced to avoid missing interesting solutions with appropriate number of function evaluations.Image tools allow us to evaluate the objective function in regions in place of points and provide an effective way to evaluate the forward and backward constraints for the multi-gravity assist trajectory optimization problem.Since the interesting solutions of the interplanetary trajectory optimization problem are often clustered in a small portion of the search space rather than being overall evenly distributed,the regionwise evaluations with image tools make the little large interval with the proper Lipschitzian tolerances sampling effective.The detailed steps of the proposed method are presented and two examples including Earth Venus Mars(EVM)transfer and Earth Venus Venus Earth Jupiter Saturn(EVVEJS)transfer are given.Finally,a comparison with solutions given by the literature demonstrates the effectiveness of the proposed method.
文摘Aimed at the great computing complexity of optimal brain surgeon (OBS) process, a pruning algorithm with penalty OBS process is presented. Compared with sensitive and regularized methods, the penalty OBS algorithm not only avoids time-consuming defect and low pruning efficiency in OBS process, but also keeps higher generalization and pruning accuracy than Levenberg-Marquardt method.
文摘Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.
文摘针对快速扩展随机树(Rapidly-exploring Random Tree,RRT)算法及其衍生算法路径规划时间长且规划效率低的问题,提出RRT算法与人工势场法结合的PAAPF-RRT机械臂路径规划算法,旨在最短的时间、最小的迭代次数内,在静态环境中找到连接起始点与终点的最优路径。首先,引入基于地图障碍物分布评估策略和采样区域优化策略,根据地图的障碍物分布、数量调整算法的步长以及偏向概率。然后,伴随随机树的生长,更新随机点的采样区域,保证随机树向目标点生长。其次,将RRT算法与人工势场法结合,当随机树与障碍物发生碰撞时,使用人工势场法引导随机树节点生长避开障碍物,解决了RRT算法随机树生长到障碍物附近且朝目标点生长的方向被障碍物遮挡时随机树无法生长的问题。最后,利用节点修剪策略,把算法生成的初始路径中的冗余节点进行修剪,得到拐点更少、路径更简洁的优化路径。实验结果表明,PAAPF-RRT算法在路径规划时间上对于RRT算法、GB-RRT算法以及RRT*算法分别减少了93.64%、73.58%、93.28%,在迭代次数方面分别下降了91.40%、79.64%、90.58%,在路径长度方面只占其他3种算法的79.34%、86.21%、95.58%。
