Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adja...Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.展开更多
Building integrated energy systems(BIESs)are pivotal for enhancing energy efficiency by accounting for a significant proportion of global energy consumption.Two key barriers that reduce the BIES operational efficiency...Building integrated energy systems(BIESs)are pivotal for enhancing energy efficiency by accounting for a significant proportion of global energy consumption.Two key barriers that reduce the BIES operational efficiency mainly lie in the renewable generation uncertainty and operational non-convexity of combined heat and power(CHP)units.To this end,this paper proposes a soft actor-critic(SAC)algorithm to solve the scheduling problem of BIES,which overcomes the model non-convexity and shows advantages in robustness and generalization.This paper also adopts a temporal fusion transformer(TFT)to enhance the optimal solution for the SAC algorithm by forecasting the renewable generation and energy demand.The TFT can effectively capture the complex temporal patterns and dependencies that span multiple steps.Furthermore,its forecasting results are interpretable due to the employment of a self-attention layer so as to assist in more trustworthy decision-making in the SAC algorithm.The proposed hybrid data-driven approach integrating TFT and SAC algorithm,i.e.,TFT-SAC approach,is trained and tested on a real-world dataset to validate its superior performance in reducing the energy cost and computational time compared with the benchmark approaches.The generalization performance for the scheduling policy,as well as the sensitivity analysis,are examined in the case studies.展开更多
Peer-to-peer(P2P)energy trading in active distribution networks(ADNs)plays a pivotal role in promoting the efficient consumption of renewable energy sources.However,it is challenging to effectively coordinate the powe...Peer-to-peer(P2P)energy trading in active distribution networks(ADNs)plays a pivotal role in promoting the efficient consumption of renewable energy sources.However,it is challenging to effectively coordinate the power dispatch of ADNs and P2P energy trading while preserving the privacy of different physical interests.Hence,this paper proposes a soft actor-critic algorithm incorporating distributed trading control(SAC-DTC)to tackle the optimal power dispatch of ADNs and the P2P energy trading considering privacy preservation among prosumers.First,the soft actor-critic(SAC)algorithm is used to optimize the control strategy of device in ADNs to minimize the operation cost,and the primary environmental information of the ADN at this point is published to prosumers.Then,a distributed generalized fast dual ascent method is used to iterate the trading process of prosumers and maximize their revenues.Subsequently,the results of trading are encrypted based on the differential privacy technique and returned to the ADN.Finally,the social welfare value consisting of ADN operation cost and P2P market revenue is utilized as a reward value to update network parameters and control strategies of the deep reinforcement learning.Simulation results show that the proposed SAC-DTC algorithm reduces the ADN operation cost,boosts the P2P market revenue,maximizes the social welfare,and exhibits high computational accuracy,demonstrating its practical application to the operation of power systems and power markets.展开更多
虚拟电厂(virtual power plant,VPP)作为多能流互联的综合能源网络,已成为中国加速实现双碳目标的重要角色。但VPP内部资源协同低碳调度面临多能流的耦合程度紧密、传统碳交易模型参数主观性强、含高维动态参数的优化目标在线求解困难...虚拟电厂(virtual power plant,VPP)作为多能流互联的综合能源网络,已成为中国加速实现双碳目标的重要角色。但VPP内部资源协同低碳调度面临多能流的耦合程度紧密、传统碳交易模型参数主观性强、含高维动态参数的优化目标在线求解困难等问题。针对这些问题,文中提出一种融合注意力机制(attention mechanism,AM)与柔性动作评价(soft actor-critic,SAC)算法的VPP多能流低碳调度方法。首先,根据VPP的随机碳流特性,面向动态参数建立基于贝叶斯优化的改进阶梯型碳交易机制。接着,以经济效益和碳排放量为目标函数构建含氢VPP多能流解耦模型。然后,考虑到该模型具有高维非线性与权重参数实时更新的特征,利用融合AM的改进SAC深度强化学习算法在连续动作空间对模型进行求解。最后,对多能流调度结果进行仿真分析和对比实验,验证了文中方法的可行性及其相较于原SAC算法较高的决策准确性。展开更多
为解决由于固定温度SAC(Soft Actor Critic)算法中存在的Q函数高估可能会导致算法陷入局部最优的问题,通过深入分析提出了一个稳定且受限的SAC算法(SCSAC:Stable Constrained Soft Actor Critic)。该算法通过改进最大熵目标函数修复固...为解决由于固定温度SAC(Soft Actor Critic)算法中存在的Q函数高估可能会导致算法陷入局部最优的问题,通过深入分析提出了一个稳定且受限的SAC算法(SCSAC:Stable Constrained Soft Actor Critic)。该算法通过改进最大熵目标函数修复固定温度SAC算法中的Q函数高估问题,同时增强算法在测试过程中稳定性的效果。最后,在4个OpenAI Gym Mujoco环境下对SCSAC算法进行了验证,实验结果表明,稳定且受限的SAC算法相比固定温度SAC算法可以有效减小Q函数高估出现的次数并能在测试中获得更加稳定的结果。展开更多
With the increasing integration of large-scale distributed energy resources into the grid,traditional distribution network optimization and dispatch methods struggle to address the challenges posed by both generation ...With the increasing integration of large-scale distributed energy resources into the grid,traditional distribution network optimization and dispatch methods struggle to address the challenges posed by both generation and load.Accounting for these issues,this paper proposes a multi-timescale coordinated optimization dispatch method for distribution networks.First,the probability box theory was employed to determine the uncertainty intervals of generation and load forecasts,based on which,the requirements for flexibility dispatch and capacity constraints of the grid were calculated and analyzed.Subsequently,a multi-timescale optimization framework was constructed,incorporating the generation and load forecast uncertainties.This framework included optimization models for dayahead scheduling,intra-day optimization,and real-time adjustments,aiming to meet flexibility needs across different timescales and improve the economic efficiency of the grid.Furthermore,an improved soft actor-critic algorithm was introduced to enhance the uncertainty exploration capability.Utilizing a centralized training and decentralized execution framework,a multi-agent SAC network model was developed to improve the decision-making efficiency of the agents.Finally,the effectiveness and superiority of the proposed method were validated using a modified IEEE-33 bus test system.展开更多
In order to solve the control problem of multiple-input multiple-output(MIMO)systems in complex and variable control environments,a model-free adaptive LSAC-PID method based on deep reinforcement learning(RL)is propos...In order to solve the control problem of multiple-input multiple-output(MIMO)systems in complex and variable control environments,a model-free adaptive LSAC-PID method based on deep reinforcement learning(RL)is proposed in this paper for automatic control of mobile robots.According to the environmental feedback,the RL agent of the upper controller outputs the optimal parameters to the lower MIMO PID controllers,which can realize the real-time PID optimal control.First,a model-free adaptive MIMO PID hybrid control strategy is presented to realize real-time optimal tuning of control parameters in terms of soft-actor-critic(SAC)algorithm,which is state-of-the-art RL algorithm.Second,in order to improve the RL convergence speed and the control performance,a Lyapunov-based reward shaping method for off-policy RL algorithm is designed,and a self-adaptive LSAC-PID tuning approach with Lyapunov-based reward is then determined.Through the policy evaluation and policy improvement of the soft policy iteration,the convergence and optimality of the proposed LSAC-PID algorithm are proved mathematically.Finally,based on the proposed reward shaping method,the reward function is designed to improve the system stability for the line-following robot.The simulation and experiment results show that the proposed adaptive LSAC-PID approach has good control performance such as fast convergence speed,high generalization and high real-time performance,and achieves real-time optimal tuning of MIMO PID parameters without the system model and control loop decoupling.展开更多
Handling ralely happened safety-critical events remains a central challenge in developing reliable,learning-driven automated driving technologies.Existing reinforcement learning(RL)models,especially those trained on n...Handling ralely happened safety-critical events remains a central challenge in developing reliable,learning-driven automated driving technologies.Existing reinforcement learning(RL)models,especially those trained on normal driving data,often lack robustness when encountering uncovered scenarios,such as sudden full braking by the lead vehicle,due to the underrepresentation of these events in training data.This study addresses this gap by introducing a Double Soft Actor-Critic(Double SAC)framework tailored for car-following control in both routine and emergency situations,with distinct reward functions designed for each scenario.A unique stochastic training environment is proposed to incorporate simulated emergency braking scenarios,enhancing model resilience in diverse and challenging conditions.Simulation results demonstrate that the Double SAC algorithm outperforms traditional methods in key metrics such as safety,efficiency,and passenger comfort,especially in emergency braking situations.Additionally,with an increase in CAV penetration rates,the stability of platoon control is greatly improved,traffic oscillations are reduced,and the system exhibits enhanced cooperative control capabilities.展开更多
基金supported by National Natural Science Foundation of China(52222215, 52272420, 52072051)。
文摘Parking in a small parking lot within limited space poses a difficult task. It often leads to deviations between the final parking posture and the target posture. These deviations can lead to partial occupancy of adjacent parking lots, which poses a safety threat to vehicles parked in these parking lots. However, previous studies have not addressed this issue. In this paper, we aim to evaluate the impact of parking deviation of existing vehicles next to the target parking lot(PDEVNTPL) on the automatic ego vehicle(AEV) parking, in terms of safety, comfort, accuracy, and efficiency of parking. A segmented parking training framework(SPTF) based on soft actor-critic(SAC) is proposed to improve parking performance. In the proposed method, the SAC algorithm incorporates strategy entropy into the objective function, to enable the AEV to learn parking strategies based on a more comprehensive understanding of the environment. Additionally, the SPTF simplifies complex parking tasks to maintain the high performance of deep reinforcement learning(DRL). The experimental results reveal that the PDEVNTPL has a detrimental influence on the AEV parking in terms of safety, accuracy, and comfort, leading to reductions of more than 27%, 54%, and 26%respectively. However, the SAC-based SPTF effectively mitigates this impact, resulting in a considerable increase in the parking success rate from 71% to 93%. Furthermore, the heading angle deviation is significantly reduced from 2.25 degrees to 0.43degrees.
文摘Building integrated energy systems(BIESs)are pivotal for enhancing energy efficiency by accounting for a significant proportion of global energy consumption.Two key barriers that reduce the BIES operational efficiency mainly lie in the renewable generation uncertainty and operational non-convexity of combined heat and power(CHP)units.To this end,this paper proposes a soft actor-critic(SAC)algorithm to solve the scheduling problem of BIES,which overcomes the model non-convexity and shows advantages in robustness and generalization.This paper also adopts a temporal fusion transformer(TFT)to enhance the optimal solution for the SAC algorithm by forecasting the renewable generation and energy demand.The TFT can effectively capture the complex temporal patterns and dependencies that span multiple steps.Furthermore,its forecasting results are interpretable due to the employment of a self-attention layer so as to assist in more trustworthy decision-making in the SAC algorithm.The proposed hybrid data-driven approach integrating TFT and SAC algorithm,i.e.,TFT-SAC approach,is trained and tested on a real-world dataset to validate its superior performance in reducing the energy cost and computational time compared with the benchmark approaches.The generalization performance for the scheduling policy,as well as the sensitivity analysis,are examined in the case studies.
基金supported by the National Natural Science Foundation of China(No.52177085).
文摘Peer-to-peer(P2P)energy trading in active distribution networks(ADNs)plays a pivotal role in promoting the efficient consumption of renewable energy sources.However,it is challenging to effectively coordinate the power dispatch of ADNs and P2P energy trading while preserving the privacy of different physical interests.Hence,this paper proposes a soft actor-critic algorithm incorporating distributed trading control(SAC-DTC)to tackle the optimal power dispatch of ADNs and the P2P energy trading considering privacy preservation among prosumers.First,the soft actor-critic(SAC)algorithm is used to optimize the control strategy of device in ADNs to minimize the operation cost,and the primary environmental information of the ADN at this point is published to prosumers.Then,a distributed generalized fast dual ascent method is used to iterate the trading process of prosumers and maximize their revenues.Subsequently,the results of trading are encrypted based on the differential privacy technique and returned to the ADN.Finally,the social welfare value consisting of ADN operation cost and P2P market revenue is utilized as a reward value to update network parameters and control strategies of the deep reinforcement learning.Simulation results show that the proposed SAC-DTC algorithm reduces the ADN operation cost,boosts the P2P market revenue,maximizes the social welfare,and exhibits high computational accuracy,demonstrating its practical application to the operation of power systems and power markets.
文摘虚拟电厂(virtual power plant,VPP)作为多能流互联的综合能源网络,已成为中国加速实现双碳目标的重要角色。但VPP内部资源协同低碳调度面临多能流的耦合程度紧密、传统碳交易模型参数主观性强、含高维动态参数的优化目标在线求解困难等问题。针对这些问题,文中提出一种融合注意力机制(attention mechanism,AM)与柔性动作评价(soft actor-critic,SAC)算法的VPP多能流低碳调度方法。首先,根据VPP的随机碳流特性,面向动态参数建立基于贝叶斯优化的改进阶梯型碳交易机制。接着,以经济效益和碳排放量为目标函数构建含氢VPP多能流解耦模型。然后,考虑到该模型具有高维非线性与权重参数实时更新的特征,利用融合AM的改进SAC深度强化学习算法在连续动作空间对模型进行求解。最后,对多能流调度结果进行仿真分析和对比实验,验证了文中方法的可行性及其相较于原SAC算法较高的决策准确性。
文摘为解决由于固定温度SAC(Soft Actor Critic)算法中存在的Q函数高估可能会导致算法陷入局部最优的问题,通过深入分析提出了一个稳定且受限的SAC算法(SCSAC:Stable Constrained Soft Actor Critic)。该算法通过改进最大熵目标函数修复固定温度SAC算法中的Q函数高估问题,同时增强算法在测试过程中稳定性的效果。最后,在4个OpenAI Gym Mujoco环境下对SCSAC算法进行了验证,实验结果表明,稳定且受限的SAC算法相比固定温度SAC算法可以有效减小Q函数高估出现的次数并能在测试中获得更加稳定的结果。
基金funded by Jilin Province Science and Technology Development Plan Project,grant number 20220203163SF.
文摘With the increasing integration of large-scale distributed energy resources into the grid,traditional distribution network optimization and dispatch methods struggle to address the challenges posed by both generation and load.Accounting for these issues,this paper proposes a multi-timescale coordinated optimization dispatch method for distribution networks.First,the probability box theory was employed to determine the uncertainty intervals of generation and load forecasts,based on which,the requirements for flexibility dispatch and capacity constraints of the grid were calculated and analyzed.Subsequently,a multi-timescale optimization framework was constructed,incorporating the generation and load forecast uncertainties.This framework included optimization models for dayahead scheduling,intra-day optimization,and real-time adjustments,aiming to meet flexibility needs across different timescales and improve the economic efficiency of the grid.Furthermore,an improved soft actor-critic algorithm was introduced to enhance the uncertainty exploration capability.Utilizing a centralized training and decentralized execution framework,a multi-agent SAC network model was developed to improve the decision-making efficiency of the agents.Finally,the effectiveness and superiority of the proposed method were validated using a modified IEEE-33 bus test system.
基金the National Key R&D Program of China(No.2018YFB1308400)。
文摘In order to solve the control problem of multiple-input multiple-output(MIMO)systems in complex and variable control environments,a model-free adaptive LSAC-PID method based on deep reinforcement learning(RL)is proposed in this paper for automatic control of mobile robots.According to the environmental feedback,the RL agent of the upper controller outputs the optimal parameters to the lower MIMO PID controllers,which can realize the real-time PID optimal control.First,a model-free adaptive MIMO PID hybrid control strategy is presented to realize real-time optimal tuning of control parameters in terms of soft-actor-critic(SAC)algorithm,which is state-of-the-art RL algorithm.Second,in order to improve the RL convergence speed and the control performance,a Lyapunov-based reward shaping method for off-policy RL algorithm is designed,and a self-adaptive LSAC-PID tuning approach with Lyapunov-based reward is then determined.Through the policy evaluation and policy improvement of the soft policy iteration,the convergence and optimality of the proposed LSAC-PID algorithm are proved mathematically.Finally,based on the proposed reward shaping method,the reward function is designed to improve the system stability for the line-following robot.The simulation and experiment results show that the proposed adaptive LSAC-PID approach has good control performance such as fast convergence speed,high generalization and high real-time performance,and achieves real-time optimal tuning of MIMO PID parameters without the system model and control loop decoupling.
基金supported in part by the lconic Specialised Cultivation Project of Yanshan University(Grant No.2022BZZD005)the Hebei Natural Science Foundation Grant No.F2024203083.
文摘Handling ralely happened safety-critical events remains a central challenge in developing reliable,learning-driven automated driving technologies.Existing reinforcement learning(RL)models,especially those trained on normal driving data,often lack robustness when encountering uncovered scenarios,such as sudden full braking by the lead vehicle,due to the underrepresentation of these events in training data.This study addresses this gap by introducing a Double Soft Actor-Critic(Double SAC)framework tailored for car-following control in both routine and emergency situations,with distinct reward functions designed for each scenario.A unique stochastic training environment is proposed to incorporate simulated emergency braking scenarios,enhancing model resilience in diverse and challenging conditions.Simulation results demonstrate that the Double SAC algorithm outperforms traditional methods in key metrics such as safety,efficiency,and passenger comfort,especially in emergency braking situations.Additionally,with an increase in CAV penetration rates,the stability of platoon control is greatly improved,traffic oscillations are reduced,and the system exhibits enhanced cooperative control capabilities.
