Wireless Sensor Networks(WSNs)play a crucial role in numerous Internet of Things(IoT)applications and next-generation communication systems,yet they continue to face challenges in balancing energy efficiency and relia...Wireless Sensor Networks(WSNs)play a crucial role in numerous Internet of Things(IoT)applications and next-generation communication systems,yet they continue to face challenges in balancing energy efficiency and reliable connectivity.This study proposes SAC-HTC(Soft Actor-Critic-based High-performance Topology Control),a deep reinforcement learning(DRL)method based on the Actor-Critic framework,implemented within a Software Defined Wireless Sensor Network(SDWSN)architecture.In this approach,sensor nodes periodically transmit state information,including coordinates,node degree,transmission power,and neighbor lists,to a centralized controller.The controller acts as the reinforcement learning(RL)agent,with the Actor generating decisions to adjust transmission ranges,while the Critic evaluates action values to reflect the overall network performance.The bidirectional Node-Controller feedback mechanism enables the controller to issue appropriate control commands to each node,ensuring the maintenance of the desired node degree,reducing energy consumption,and preserving network connectivity.The algorithmfurther incorporates soft entropy adjustment to balance exploration and exploitation,alongwith an off-policy mechanism for efficient data reuse,making it well-suited to the resource-constrained conditions ofWSNs.Simulation results demonstrate that SAC-HTC not only outperforms traditional methods and several existing RL algorithms but also achieves faster convergence,optimized communication range control,global connectivity maintenance,and extended network lifetime.The key novelty of this research lies in the integration of the SAC method with the SDWSN architecture forWSNs topology control,providing an adaptive,efficient,and highly promisingmechanism for large-scale,dynamic,and high-performance sensor networks.展开更多
In order to address the issue of overly conservative offline reinforcement learning(RL) methods that limit the generalization of policy in the out-of-distribution(OOD) region,this article designs a surrogate target fo...In order to address the issue of overly conservative offline reinforcement learning(RL) methods that limit the generalization of policy in the out-of-distribution(OOD) region,this article designs a surrogate target for OOD value function based on dataset distance and proposes a novel generalized Q-learning mechanism with distance regularization(GQDR).In theory,we not only prove the convergence of GQDR,but also ensure that the difference between the Q-value learned by GQDR and its true value is bounded.Furthermore,an offline generalized actor-critic method with distance regularization(OGACDR) is proposed by combining GQDR with actor-critic learning framework.Two implementations of OGACDR,OGACDR-EXP and OGACDRSQR,are introduced according to exponential(EXP) and opensquare(SQR) distance weight functions,and it has been theoretically proved that OGACDR provides a safe policy improvement.Experimental results on Gym-MuJoCo continuous control tasks show that OGACDR can not only alleviate the overestimation and overconservatism of Q-value function,but also outperform conservative offline RL baselines.展开更多
With the rapid development of artificial intelligence,intelligent air combat maneuver decision-making(ACMD)has garnered global attention.Although deep reinforcement learning provides a promising approach to ACMD,exist...With the rapid development of artificial intelligence,intelligent air combat maneuver decision-making(ACMD)has garnered global attention.Although deep reinforcement learning provides a promising approach to ACMD,existing methods often suffer from rigid reward functions and limited adaptability to evolving adversarial strategies.Moreover,most research assumes open airspace,overlooking the influence of potential obstacles.In this paper,we address one-on-one within-visual-range ACMD in obstructed environments,and propose an improved Soft Actor-Critic(SAC)algorithm trained under a curriculum self-play framework.A maneuver strategy mirroring inference module is integrated to estimate each other's likely positions when visual obstruction occurs.By leveraging curriculum learning to guide progressive experience accumulation and self-play for adversarial evolution,our method enhances both training efficiency and tactical diversity.We further integrate an attention mechanism that dynamically adjusts the weights of sub-rewards,enabling the learned policy to adapt to rapidly changing air combat situations.Numerical simulations demonstrate that our enhanced SAC converges more quickly and achieves higher win rates than other baseline methods.An animation is available at bilibili.com/video/BV1BHVszHE98 for better illustration.展开更多
Decision-making of connected and automated vehicles(CAV)includes a sequence of driving maneuvers that improve safety and efficiency,characterized by complex scenarios,strong uncertainty,and high real-time requirements...Decision-making of connected and automated vehicles(CAV)includes a sequence of driving maneuvers that improve safety and efficiency,characterized by complex scenarios,strong uncertainty,and high real-time requirements.Deep reinforcement learning(DRL)exhibits excellent capability of real-time decision-making and adaptability to complex scenarios,and generalization abilities.However,it is arduous to guarantee complete driving safety and efficiency under the constraints of training samples and costs.This paper proposes a Mixture of Expert method(MoE)based on Soft Actor-Critic(SAC),where the upper-level discriminator dynamically decides whether to activate the lower-level DRL expert or the heuristic expert based on the features of the input state.To further enhance the performance of the DRL expert,a buffer zone is introduced in the reward function,preemptively applying penalties before insecure situations occur.In order to minimize collision and off-road rates,the Intelligent Driver Model(IDM)and Minimizing Overall Braking Induced by Lane changes(MOBIL)strategy are designed by heuristic experts.Finally,tested in typical simulation scenarios,MOE shows a 13.75%improvement in driving efficiency compared with the traditional DRL method with continuous action space.It ensures high safety with zero collision and zero off-road rates while maintaining high adaptability.展开更多
随着可再生能源并网发电量的不断增加,由电力电子设备引发的电力系统次同步振荡问题逐渐凸显,给电力系统的安全稳定运行带来了新的挑战。除此之外,当目标电力系统规模较大时,常用的基于线性化模型的分析方法面临着维数灾难。为了解决上...随着可再生能源并网发电量的不断增加,由电力电子设备引发的电力系统次同步振荡问题逐渐凸显,给电力系统的安全稳定运行带来了新的挑战。除此之外,当目标电力系统规模较大时,常用的基于线性化模型的分析方法面临着维数灾难。为了解决上述问题,根据强化学习原理,通过动作-评价(Actor-Critic)学习框架提出一种对风机换流器控制参数的优化策略。通过搜集永磁直驱风机(permanent magnetic synchronous generator,PMSG)运行状态数据,训练强化学习代理(Agent),以此评估风机运行状态及其稳定性,并寻找优化风机换流器参数的最优策略。该训练方法得到的代理能够基于时域采样数据对风机换流器参数进行优化,从而有效抑制由于换流器诱发的振荡现象,在没有建立线性化分析模型的情况下,能够有效优化并增强电力系统的稳定性。经实验验证,该优化策略在采样数据有噪声干扰的情况下仍然具有良好的优化性能。展开更多
文摘Wireless Sensor Networks(WSNs)play a crucial role in numerous Internet of Things(IoT)applications and next-generation communication systems,yet they continue to face challenges in balancing energy efficiency and reliable connectivity.This study proposes SAC-HTC(Soft Actor-Critic-based High-performance Topology Control),a deep reinforcement learning(DRL)method based on the Actor-Critic framework,implemented within a Software Defined Wireless Sensor Network(SDWSN)architecture.In this approach,sensor nodes periodically transmit state information,including coordinates,node degree,transmission power,and neighbor lists,to a centralized controller.The controller acts as the reinforcement learning(RL)agent,with the Actor generating decisions to adjust transmission ranges,while the Critic evaluates action values to reflect the overall network performance.The bidirectional Node-Controller feedback mechanism enables the controller to issue appropriate control commands to each node,ensuring the maintenance of the desired node degree,reducing energy consumption,and preserving network connectivity.The algorithmfurther incorporates soft entropy adjustment to balance exploration and exploitation,alongwith an off-policy mechanism for efficient data reuse,making it well-suited to the resource-constrained conditions ofWSNs.Simulation results demonstrate that SAC-HTC not only outperforms traditional methods and several existing RL algorithms but also achieves faster convergence,optimized communication range control,global connectivity maintenance,and extended network lifetime.The key novelty of this research lies in the integration of the SAC method with the SDWSN architecture forWSNs topology control,providing an adaptive,efficient,and highly promisingmechanism for large-scale,dynamic,and high-performance sensor networks.
基金supported by the National Natural Science Foundation of China(62373364,62176259)the Key Research and Development Program of Jiangsu Province(BE2022095)。
文摘In order to address the issue of overly conservative offline reinforcement learning(RL) methods that limit the generalization of policy in the out-of-distribution(OOD) region,this article designs a surrogate target for OOD value function based on dataset distance and proposes a novel generalized Q-learning mechanism with distance regularization(GQDR).In theory,we not only prove the convergence of GQDR,but also ensure that the difference between the Q-value learned by GQDR and its true value is bounded.Furthermore,an offline generalized actor-critic method with distance regularization(OGACDR) is proposed by combining GQDR with actor-critic learning framework.Two implementations of OGACDR,OGACDR-EXP and OGACDRSQR,are introduced according to exponential(EXP) and opensquare(SQR) distance weight functions,and it has been theoretically proved that OGACDR provides a safe policy improvement.Experimental results on Gym-MuJoCo continuous control tasks show that OGACDR can not only alleviate the overestimation and overconservatism of Q-value function,but also outperform conservative offline RL baselines.
基金support of the National Key Research and Development Plan(No.2021YFB3302501)the financial support of the National Science Foundation of China(No.12161076)the financial support of the Fundamental Research Funds for the Central Universities(No.DUT25GF207).
文摘With the rapid development of artificial intelligence,intelligent air combat maneuver decision-making(ACMD)has garnered global attention.Although deep reinforcement learning provides a promising approach to ACMD,existing methods often suffer from rigid reward functions and limited adaptability to evolving adversarial strategies.Moreover,most research assumes open airspace,overlooking the influence of potential obstacles.In this paper,we address one-on-one within-visual-range ACMD in obstructed environments,and propose an improved Soft Actor-Critic(SAC)algorithm trained under a curriculum self-play framework.A maneuver strategy mirroring inference module is integrated to estimate each other's likely positions when visual obstruction occurs.By leveraging curriculum learning to guide progressive experience accumulation and self-play for adversarial evolution,our method enhances both training efficiency and tactical diversity.We further integrate an attention mechanism that dynamically adjusts the weights of sub-rewards,enabling the learned policy to adapt to rapidly changing air combat situations.Numerical simulations demonstrate that our enhanced SAC converges more quickly and achieves higher win rates than other baseline methods.An animation is available at bilibili.com/video/BV1BHVszHE98 for better illustration.
基金Supported by National Key R&D Program of China(Grant No.2022YFB2503203)National Natural Science Foundation of China(Grant No.U1964206).
文摘Decision-making of connected and automated vehicles(CAV)includes a sequence of driving maneuvers that improve safety and efficiency,characterized by complex scenarios,strong uncertainty,and high real-time requirements.Deep reinforcement learning(DRL)exhibits excellent capability of real-time decision-making and adaptability to complex scenarios,and generalization abilities.However,it is arduous to guarantee complete driving safety and efficiency under the constraints of training samples and costs.This paper proposes a Mixture of Expert method(MoE)based on Soft Actor-Critic(SAC),where the upper-level discriminator dynamically decides whether to activate the lower-level DRL expert or the heuristic expert based on the features of the input state.To further enhance the performance of the DRL expert,a buffer zone is introduced in the reward function,preemptively applying penalties before insecure situations occur.In order to minimize collision and off-road rates,the Intelligent Driver Model(IDM)and Minimizing Overall Braking Induced by Lane changes(MOBIL)strategy are designed by heuristic experts.Finally,tested in typical simulation scenarios,MOE shows a 13.75%improvement in driving efficiency compared with the traditional DRL method with continuous action space.It ensures high safety with zero collision and zero off-road rates while maintaining high adaptability.
文摘随着可再生能源并网发电量的不断增加,由电力电子设备引发的电力系统次同步振荡问题逐渐凸显,给电力系统的安全稳定运行带来了新的挑战。除此之外,当目标电力系统规模较大时,常用的基于线性化模型的分析方法面临着维数灾难。为了解决上述问题,根据强化学习原理,通过动作-评价(Actor-Critic)学习框架提出一种对风机换流器控制参数的优化策略。通过搜集永磁直驱风机(permanent magnetic synchronous generator,PMSG)运行状态数据,训练强化学习代理(Agent),以此评估风机运行状态及其稳定性,并寻找优化风机换流器参数的最优策略。该训练方法得到的代理能够基于时域采样数据对风机换流器参数进行优化,从而有效抑制由于换流器诱发的振荡现象,在没有建立线性化分析模型的情况下,能够有效优化并增强电力系统的稳定性。经实验验证,该优化策略在采样数据有噪声干扰的情况下仍然具有良好的优化性能。
