Double Q-learning has been shown to be effective in reinforcement learning scenarios when the reward system is stochastic. We apply the idea of double learning that this algorithm uses to Sarsa and Expected Sarsa, pro...Double Q-learning has been shown to be effective in reinforcement learning scenarios when the reward system is stochastic. We apply the idea of double learning that this algorithm uses to Sarsa and Expected Sarsa, producing two new algorithms called Double Sarsa and Double Expected Sarsa that are shown to be more robust than their single counterparts when rewards are stochastic. We find that these algorithms add a significant amount of stability in the learning process at only a minor computational cost, which leads to higher returns when using an on-policy algorithm. We then use shallow and deep neural networks to approximate the actionvalue, and show that Double Sarsa and Double Expected Sarsa are much more stable after convergence and can collect larger rewards than the single versions.展开更多
文摘本文提出了一种基于5要素试错更新算法SARSA(λ)强化学习的随机最优自动发电控制方法.该方法不依赖任何系统模型和先验知识并通过试错机理寻求最优控制策略.以控制性能标准(control performance standards,CPS)和区域控制偏差(areal control error,ACE)瞬时滚动值为基础设计了即时奖励函数,有效提高了该方法的收敛速度和控制效果,并在算法中融入了资格迹以解决二次调频过程的延时问题.本文所提出的控制方法在进行状态空间搜索时,能有效摆脱避免搜索较大扰动状态,以此获得更佳的控制效果.标准两区域和南方电网仿真模型研究表明,本算法能给系统提供更加安全的控制策略,具有比Q(λ)算法更好的控制性能,有效提高CPS考核的合格率.
文摘Double Q-learning has been shown to be effective in reinforcement learning scenarios when the reward system is stochastic. We apply the idea of double learning that this algorithm uses to Sarsa and Expected Sarsa, producing two new algorithms called Double Sarsa and Double Expected Sarsa that are shown to be more robust than their single counterparts when rewards are stochastic. We find that these algorithms add a significant amount of stability in the learning process at only a minor computational cost, which leads to higher returns when using an on-policy algorithm. We then use shallow and deep neural networks to approximate the actionvalue, and show that Double Sarsa and Double Expected Sarsa are much more stable after convergence and can collect larger rewards than the single versions.
