The coordinated optimization problem of the electricity-gas-heat integrated energy system(IES)has the characteristics of strong coupling,non-convexity,and nonlinearity.The centralized optimization method has a high co...The coordinated optimization problem of the electricity-gas-heat integrated energy system(IES)has the characteristics of strong coupling,non-convexity,and nonlinearity.The centralized optimization method has a high cost of communication and complex modeling.Meanwhile,the traditional numerical iterative solution cannot deal with uncertainty and solution efficiency,which is difficult to apply online.For the coordinated optimization problem of the electricity-gas-heat IES in this study,we constructed a model for the distributed IES with a dynamic distribution factor and transformed the centralized optimization problem into a distributed optimization problem in the multi-agent reinforcement learning environment using multi-agent deep deterministic policy gradient.Introducing the dynamic distribution factor allows the system to consider the impact of changes in real-time supply and demand on system optimization,dynamically coordinating different energy sources for complementary utilization and effectively improving the system economy.Compared with centralized optimization,the distributed model with multiple decision centers can achieve similar results while easing the pressure on system communication.The proposed method considers the dual uncertainty of renewable energy and load in the training.Compared with the traditional iterative solution method,it can better cope with uncertainty and realize real-time decision making of the system,which is conducive to the online application.Finally,we verify the effectiveness of the proposed method using an example of an IES coupled with three energy hub agents.展开更多
With the rapid growth of connected devices,traditional edge-cloud systems are under overload pressure.Using mobile edge computing(MEC)to assist unmanned aerial vehicles(UAVs)as low altitude platform stations(LAPS)for ...With the rapid growth of connected devices,traditional edge-cloud systems are under overload pressure.Using mobile edge computing(MEC)to assist unmanned aerial vehicles(UAVs)as low altitude platform stations(LAPS)for communication and computation to build air-ground integrated networks(AGINs)offers a promising solution for seamless network coverage of remote internet of things(IoT)devices in the future.To address the performance demands of future mobile devices(MDs),we proposed an MEC-assisted AGIN system.The goal is to minimize the long-term computational overhead of MDs by jointly optimizing transmission power,flight trajecto-ries,resource allocation,and offloading ratios,while utilizing non-orthogonal multiple access(NOMA)to improve device connectivity of large-scale MDs and spectral efficiency.We first designed an adaptive clustering scheme based on K-Means to cluster MDs and established commu-nication links,improving efficiency and load balancing.Then,considering system dynamics,we introduced a partial computation offloading algorithm based on multi-agent deep deterministic pol-icy gradient(MADDPG),modeling the multi-UAV computation offloading problem as a Markov decision process(MDP).This algorithm optimizes resource allocation through centralized training and distributed execution,reducing computational overhead.Simulation results show that the pro-posed algorithm not only converges stably but also outperforms other benchmark algorithms in han-dling complex scenarios with multiple devices.展开更多
为提高多无人船编队系统的导航能力,提出了一种基于注意力机制的多智能体深度确定性策略梯度(ATMADDPG:Attention Mechanism based Multi-Agent Deep Deterministic Policy Gradient)算法。该算法在训练阶段,通过大量试验训练出最佳策略...为提高多无人船编队系统的导航能力,提出了一种基于注意力机制的多智能体深度确定性策略梯度(ATMADDPG:Attention Mechanism based Multi-Agent Deep Deterministic Policy Gradient)算法。该算法在训练阶段,通过大量试验训练出最佳策略,并在实验阶段直接使用训练出的最佳策略得到最佳编队路径。仿真实验将4艘相同的“百川号”无人船作为实验对象。实验结果表明,基于ATMADDPG算法的队形保持策略能实现稳定的多无人船编队导航,并在一定程度上满足队形保持的要求。相较于多智能体深度确定性策略梯度(MADDPG:Multi-Agent Depth Deterministic Policy Gradient)算法,所提出的ATMADDPG算法在收敛速度、队形保持能力和对环境变化的适应性等方面表现出更优越的性能,综合导航效率可提高约80%,具有较大的应用潜力。展开更多
The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents ...The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents can explore and summarize the environment to achieve autonomous deci-sions in the continuous state space and action space.In this paper,a cooperative defense with DDPG via swarms of unmanned aerial vehicle(UAV)is developed and validated,which has shown promising practical value in the effect of defending.We solve the sparse rewards problem of reinforcement learning pair in a long-term task by building the reward function of UAV swarms and optimizing the learning process of artificial neural network based on the DDPG algorithm to reduce the vibration in the learning process.The experimental results show that the DDPG algorithm can guide the UAVs swarm to perform the defense task efficiently,meeting the requirements of a UAV swarm for non-centralization,autonomy,and promoting the intelligent development of UAVs swarm as well as the decision-making process.展开更多
针对光储直流微电网易受光伏资源波动、负荷侧波动等不确定扰动影响,进而引发的直流母线电压波动问题,在传统自抗扰控制(linear active disturbance rejection control,LADRC)的基础上,提出一种参数动态协同自抗扰控制(dynamic coordina...针对光储直流微电网易受光伏资源波动、负荷侧波动等不确定扰动影响,进而引发的直流母线电压波动问题,在传统自抗扰控制(linear active disturbance rejection control,LADRC)的基础上,提出一种参数动态协同自抗扰控制(dynamic coordination of parameters for active disturbance rejection control,DCLADRC),引入两个新的观测变量并增加一维带宽参数,旨在通过深度确定性策略梯度(deterministic policy gradient,DDPG)算法动态调整两级带宽间的协调因子k,提高观测器多频域扰动下的观测精度及收敛速度,优化控制器的抗扰性,增强母线电压稳定性,从而使得储能能够更好地发挥“削峰填谷”的调节作用。物理实验结果表明,受到扰动后,对比LADRC与双闭环比例积分(double closed loop proportion-integration,Double_PI)控制,所提的DCLADRC电压偏移量分别减少了75%和83%。展开更多
为适应大容量同步发电机组并网点母线电压波动增加对自动电压调节器(automatic voltage regulator,AVR)系统响应能力的更高要求,提出一种基于含探索网络的双延迟深度确定性策略梯度(twin delayed deep deterministic policy gradient wi...为适应大容量同步发电机组并网点母线电压波动增加对自动电压调节器(automatic voltage regulator,AVR)系统响应能力的更高要求,提出一种基于含探索网络的双延迟深度确定性策略梯度(twin delayed deep deterministic policy gradient with Explorer network,TD3EN)算法的同步发电机励磁电压控制方法。首先,通过传递函数对同步发电机励磁调压子系统进行建模;然后建立TD3EN算法探索网络、动作网络和评价网络,并设置相应参数;接着利用TD3EN算法训练智能体,通过探索网络探索动作空间,并根据评价网络更新动作网络参数,使其为AVR提供控制信号;将训练完成的智能体接入AVR系统,实现对发电机机端电压的控制。仿真结果表明,所提方法提高了AVR系统响应调节指令和应对电压暂降的能力。展开更多
基金supported by The National Key R&D Program of China(2020YFB0905900):Research on artificial intelligence application of power internet of things.
文摘The coordinated optimization problem of the electricity-gas-heat integrated energy system(IES)has the characteristics of strong coupling,non-convexity,and nonlinearity.The centralized optimization method has a high cost of communication and complex modeling.Meanwhile,the traditional numerical iterative solution cannot deal with uncertainty and solution efficiency,which is difficult to apply online.For the coordinated optimization problem of the electricity-gas-heat IES in this study,we constructed a model for the distributed IES with a dynamic distribution factor and transformed the centralized optimization problem into a distributed optimization problem in the multi-agent reinforcement learning environment using multi-agent deep deterministic policy gradient.Introducing the dynamic distribution factor allows the system to consider the impact of changes in real-time supply and demand on system optimization,dynamically coordinating different energy sources for complementary utilization and effectively improving the system economy.Compared with centralized optimization,the distributed model with multiple decision centers can achieve similar results while easing the pressure on system communication.The proposed method considers the dual uncertainty of renewable energy and load in the training.Compared with the traditional iterative solution method,it can better cope with uncertainty and realize real-time decision making of the system,which is conducive to the online application.Finally,we verify the effectiveness of the proposed method using an example of an IES coupled with three energy hub agents.
基金supported by the Gansu Province Key Research and Development Plan(No.23YFGA0062)Gansu Provin-cial Innovation Fund(No.2022A-215).
文摘With the rapid growth of connected devices,traditional edge-cloud systems are under overload pressure.Using mobile edge computing(MEC)to assist unmanned aerial vehicles(UAVs)as low altitude platform stations(LAPS)for communication and computation to build air-ground integrated networks(AGINs)offers a promising solution for seamless network coverage of remote internet of things(IoT)devices in the future.To address the performance demands of future mobile devices(MDs),we proposed an MEC-assisted AGIN system.The goal is to minimize the long-term computational overhead of MDs by jointly optimizing transmission power,flight trajecto-ries,resource allocation,and offloading ratios,while utilizing non-orthogonal multiple access(NOMA)to improve device connectivity of large-scale MDs and spectral efficiency.We first designed an adaptive clustering scheme based on K-Means to cluster MDs and established commu-nication links,improving efficiency and load balancing.Then,considering system dynamics,we introduced a partial computation offloading algorithm based on multi-agent deep deterministic pol-icy gradient(MADDPG),modeling the multi-UAV computation offloading problem as a Markov decision process(MDP).This algorithm optimizes resource allocation through centralized training and distributed execution,reducing computational overhead.Simulation results show that the pro-posed algorithm not only converges stably but also outperforms other benchmark algorithms in han-dling complex scenarios with multiple devices.
基金supported by the Key Research and Development Program of Shaanxi(2022GY-089)the Natural Science Basic Research Program of Shaanxi(2022JQ-593).
文摘The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents can explore and summarize the environment to achieve autonomous deci-sions in the continuous state space and action space.In this paper,a cooperative defense with DDPG via swarms of unmanned aerial vehicle(UAV)is developed and validated,which has shown promising practical value in the effect of defending.We solve the sparse rewards problem of reinforcement learning pair in a long-term task by building the reward function of UAV swarms and optimizing the learning process of artificial neural network based on the DDPG algorithm to reduce the vibration in the learning process.The experimental results show that the DDPG algorithm can guide the UAVs swarm to perform the defense task efficiently,meeting the requirements of a UAV swarm for non-centralization,autonomy,and promoting the intelligent development of UAVs swarm as well as the decision-making process.
文摘针对光储直流微电网易受光伏资源波动、负荷侧波动等不确定扰动影响,进而引发的直流母线电压波动问题,在传统自抗扰控制(linear active disturbance rejection control,LADRC)的基础上,提出一种参数动态协同自抗扰控制(dynamic coordination of parameters for active disturbance rejection control,DCLADRC),引入两个新的观测变量并增加一维带宽参数,旨在通过深度确定性策略梯度(deterministic policy gradient,DDPG)算法动态调整两级带宽间的协调因子k,提高观测器多频域扰动下的观测精度及收敛速度,优化控制器的抗扰性,增强母线电压稳定性,从而使得储能能够更好地发挥“削峰填谷”的调节作用。物理实验结果表明,受到扰动后,对比LADRC与双闭环比例积分(double closed loop proportion-integration,Double_PI)控制,所提的DCLADRC电压偏移量分别减少了75%和83%。
文摘为适应大容量同步发电机组并网点母线电压波动增加对自动电压调节器(automatic voltage regulator,AVR)系统响应能力的更高要求,提出一种基于含探索网络的双延迟深度确定性策略梯度(twin delayed deep deterministic policy gradient with Explorer network,TD3EN)算法的同步发电机励磁电压控制方法。首先,通过传递函数对同步发电机励磁调压子系统进行建模;然后建立TD3EN算法探索网络、动作网络和评价网络,并设置相应参数;接着利用TD3EN算法训练智能体,通过探索网络探索动作空间,并根据评价网络更新动作网络参数,使其为AVR提供控制信号;将训练完成的智能体接入AVR系统,实现对发电机机端电压的控制。仿真结果表明,所提方法提高了AVR系统响应调节指令和应对电压暂降的能力。
