Bionic gait learning of quadruped robots based on reinforcement learning has become a hot research topic.The proximal policy optimization(PPO)algorithm has a low probability of learning a successful gait from scratch ...Bionic gait learning of quadruped robots based on reinforcement learning has become a hot research topic.The proximal policy optimization(PPO)algorithm has a low probability of learning a successful gait from scratch due to problems such as reward sparsity.To solve the problem,we propose a experience evolution proximal policy optimization(EEPPO)algorithm which integrates PPO with priori knowledge highlighting by evolutionary strategy.We use the successful trained samples as priori knowledge to guide the learning direction in order to increase the success probability of the learning algorithm.To verify the effectiveness of the proposed EEPPO algorithm,we have conducted simulation experiments of the quadruped robot gait learning task on Pybullet.Experimental results show that the central pattern generator based radial basis function(CPG-RBF)network and the policy network are simultaneously updated to achieve the quadruped robot’s bionic diagonal trot gait learning task using key information such as the robot’s speed,posture and joints information.Experimental comparison results with the traditional soft actor-critic(SAC)algorithm validate the superiority of the proposed EEPPO algorithm,which can learn a more stable diagonal trot gait in flat terrain.展开更多
Walking is the basic skill of a legged robot, and one of the promising ways to improve the walking performance and its adaptation to environment changes is to let the robot learn its walking by itself. Currently, most...Walking is the basic skill of a legged robot, and one of the promising ways to improve the walking performance and its adaptation to environment changes is to let the robot learn its walking by itself. Currently, most of the walking learning methods are based on robot vision system or some external sensing equipment to estimate the walking performance of certain walking parameters, and therefore are usually only applicable under laboratory condition, where environment can be pre-defined. Inspired by the rhythmic swing movement during walking of legged animals and the behavior of their adjusting their walking gait on different walking surfaces, a concept of walking rhythmic pattern（WRP） is proposed to evaluate the walking specialty of legged robot, which is just based on the walking dynamics of the robot. Based on the onboard acceleration sensor data, a method to calculate WRP using power spectrum in frequency domain and diverse smooth filters is also presented. Since the evaluation of WRP is only based on the walking dynamics data of the robot＇s body, the proposed method doesn＇t require prior knowledge of environment and thus can be applied in unknown environment. A gait learning approach of legged robots based on WRP and evolution algorithm（EA） is introduced. By using the proposed approach, a quadruped robot can learn its locomotion by its onboard sensing in an unknown environment, where the robot has no prior knowledge about this place. The experimental result proves proportional relationship exits between WRP match score and walking performance of legged robot, which can be used to evaluate the walking performance in walking optimization under unknown environment.展开更多
基金the National Natural Science Foundation of China(No.62103009)。
文摘Bionic gait learning of quadruped robots based on reinforcement learning has become a hot research topic.The proximal policy optimization(PPO)algorithm has a low probability of learning a successful gait from scratch due to problems such as reward sparsity.To solve the problem,we propose a experience evolution proximal policy optimization(EEPPO)algorithm which integrates PPO with priori knowledge highlighting by evolutionary strategy.We use the successful trained samples as priori knowledge to guide the learning direction in order to increase the success probability of the learning algorithm.To verify the effectiveness of the proposed EEPPO algorithm,we have conducted simulation experiments of the quadruped robot gait learning task on Pybullet.Experimental results show that the central pattern generator based radial basis function(CPG-RBF)network and the policy network are simultaneously updated to achieve the quadruped robot’s bionic diagonal trot gait learning task using key information such as the robot’s speed,posture and joints information.Experimental comparison results with the traditional soft actor-critic(SAC)algorithm validate the superiority of the proposed EEPPO algorithm,which can learn a more stable diagonal trot gait in flat terrain.
基金supported by National Hi-tech Research and Development Program of China (863 Program, Grant No. 2009AA04Z213)"Dawn Tracking" Program of Shanghai Education Commission, China (Grant No. 10GG11)+1 种基金International Technology Co-operation Project (Grant No. 2010DFA12210)Shanghai Science and Technology Committee Talent Program of China (Grant No. 11XD1404800)
文摘Walking is the basic skill of a legged robot, and one of the promising ways to improve the walking performance and its adaptation to environment changes is to let the robot learn its walking by itself. Currently, most of the walking learning methods are based on robot vision system or some external sensing equipment to estimate the walking performance of certain walking parameters, and therefore are usually only applicable under laboratory condition, where environment can be pre-defined. Inspired by the rhythmic swing movement during walking of legged animals and the behavior of their adjusting their walking gait on different walking surfaces, a concept of walking rhythmic pattern（WRP） is proposed to evaluate the walking specialty of legged robot, which is just based on the walking dynamics of the robot. Based on the onboard acceleration sensor data, a method to calculate WRP using power spectrum in frequency domain and diverse smooth filters is also presented. Since the evaluation of WRP is only based on the walking dynamics data of the robot＇s body, the proposed method doesn＇t require prior knowledge of environment and thus can be applied in unknown environment. A gait learning approach of legged robots based on WRP and evolution algorithm（EA） is introduced. By using the proposed approach, a quadruped robot can learn its locomotion by its onboard sensing in an unknown environment, where the robot has no prior knowledge about this place. The experimental result proves proportional relationship exits between WRP match score and walking performance of legged robot, which can be used to evaluate the walking performance in walking optimization under unknown environment.
