The rudder mechanism of the X-rudder autonomous underwater cehicle(AUV)is relatively complex,and fault diagnosis capability is an important guarantee for its task execution in complex underwater environments.However,t...The rudder mechanism of the X-rudder autonomous underwater cehicle(AUV)is relatively complex,and fault diagnosis capability is an important guarantee for its task execution in complex underwater environments.However,traditional fault diagnosis methods currently rely on prior knowledge and expert experience,and lack accuracy.In order to improve the autonomy and accuracy of fault diagnosis methods,and overcome the shortcomings of traditional algorithms,this paper proposes an X-steering AUV fault diagnosis model based on the deep reinforcement learning deep Q network(DQN)algorithm,which can learn the relationship between state data and fault types,map raw residual data to corresponding fault patterns,and achieve end-to-end mapping.In addition,to solve the problem of few X-steering fault sample data,Dropout technology is introduced during the model training phase to improve the performance of the DQN algorithm.Experimental results show that the proposed model has improved the convergence speed and comprehensive performance indicators compared to the unimproved DQN algorithm,with precision,recall,F_(1-score),and accuracy reaching up to 100%,98.07%,99.02%,and 98.50% respectively,and the model’s accuracy is higher than other machine learning algorithms like back propagation,support vector machine.展开更多
Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment.Reinforcement Learning methods enable a st...Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment.Reinforcement Learning methods enable a state action function in mobile robots suited to their environment.During trial-and-error interaction with its surroundings,it helps a robot tofind an ideal behavior on its own.The Deep Q Network(DQN)algorithm is used in TurtleBot 3(TB3)to achieve the goal by successfully avoiding the obstacles.But it requires a large number of training iterations.This research mainly focuses on a mobility robot’s best path prediction utilizing DQN and the Artificial Potential Field(APF)algorithms.First,a TB3 Waffle Pi DQN is built and trained to reach the goal.Then the APF shortest path algorithm is incorporated into the DQN algorithm.The proposed planning approach is compared with the standard DQN method in a virtual environment based on the Robot Operation System(ROS).The results from the simulation show that the combination is effective for DQN and APF gives a better optimal path and takes less time when compared to the conventional DQN algo-rithm.The performance improvement rate of the proposed DQN+APF in comparison with DQN in terms of the number of successful targets is attained by 88%.The performance of the proposed DQN+APF in comparison with DQN in terms of average time is achieved by 0.331 s.The performance of the proposed DQN+APF in comparison with DQN average rewards in which the positive goal is attained by 85%and the negative goal is attained by-90%.展开更多
Reinforcement Learning(RL)techniques are being studied to solve the Demand and Capacity Balancing(DCB)problems to fully exploit their computational performance.A locally gen-eralised Multi-Agent Reinforcement Learning...Reinforcement Learning(RL)techniques are being studied to solve the Demand and Capacity Balancing(DCB)problems to fully exploit their computational performance.A locally gen-eralised Multi-Agent Reinforcement Learning(MARL)for real-world DCB problems is proposed.The proposed method can deploy trained agents directly to unseen scenarios in a specific Air Traffic Flow Management(ATFM)region to quickly obtain a satisfactory solution.In this method,agents of all flights in a scenario form a multi-agent decision-making system based on partial observation.The trained agent with the customised neural network can be deployed directly on the corresponding flight,allowing it to solve the DCB problem jointly.A cooperation coefficient is introduced in the reward function,which is used to adjust the agent’s cooperation preference in a multi-agent system,thereby controlling the distribution of flight delay time allocation.A multi-iteration mechanism is designed for the DCB decision-making framework to deal with problems arising from non-stationarity in MARL and to ensure that all hotspots are eliminated.Experiments based on large-scale high-complexity real-world scenarios are conducted to verify the effectiveness and efficiency of the method.From a statis-tical point of view,it is proven that the proposed method is generalised within the scope of the flights and sectors of interest,and its optimisation performance outperforms the standard computer-assisted slot allocation and state-of-the-art RL-based DCB methods.The sensitivity analysis preliminarily reveals the effect of the cooperation coefficient on delay time allocation.展开更多
By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning...By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning of mobile robots.However,the traditional DDQN algorithm suffers from sparse rewards and inefficient utilization of high-quality data.Targeting those problems,an improved DDQN algorithm based on average Q-value estimation and reward redistribution was proposed.First,to enhance the precision of the target Q-value,the average of multiple previously learned Q-values from the target Q network is used to replace the single Q-value from the current target Q network.Next,a reward redistribution mechanism is designed to overcome the sparse reward problem by adjusting the final reward of each action using the round reward from trajectory information.Additionally,a reward-prioritized experience selection method is introduced,which ranks experience samples according to reward values to ensure frequent utilization of high-quality data.Finally,simulation experiments are conducted to verify the effectiveness of the proposed algorithm in fixed-position scenario and random environments.The experimental results show that compared to the traditional DDQN algorithm,the proposed algorithm achieves shorter average running time,higher average return and fewer average steps.The performance of the proposed algorithm is improved by 11.43%in the fixed scenario and 8.33%in random environments.It not only plans economic and safe paths but also significantly improves efficiency and generalization in path planning,making it suitable for widespread application in autonomous navigation and industrial automation.展开更多
A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture ...A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture adjustment. A robot is taken as an agent and trained to walk steadily on an uneven surface with obstacles, using a simple reward function based on forward progress. The reward-punishment (RP) mechanism of the DQN algorithm is established after obtaining the offline gait which was generated in advance foot trajectory planning. Instead of implementing a complex dynamic model, the proposed method enables the biped robot to learn to adjust its posture on the uneven ground and ensures walking stability. The performance and effectiveness of the proposed algorithm was validated in the V-REP simulation environment. The results demonstrate that the biped robot's lateral tile angle is less than 3° after implementing the proposed method and the walking stability is obviously improved.展开更多
Multi-agent mobile applications play an essential role in mobile applications and have attracted more and more researchers’attention.Previous work has always focused on multi-agent applications with perfect informati...Multi-agent mobile applications play an essential role in mobile applications and have attracted more and more researchers’attention.Previous work has always focused on multi-agent applications with perfect information.Researchers are usually based on human-designed rules to provide decision-making searching services.However,existing methods for solving perfect-information mobile applications cannot be directly applied to imperfect-information mobile applications.Here,we take the Contact Bridge,a multi-agent application with imperfect information,for the case study.We propose an enhanced searching strategy to deal with multi-agent applications with imperfect information.We design a self-training bidding system model and apply a Recurrent Neural Network(RNN)to model the bidding process.The bridge system model consists of two parts,a bidding prediction system based on imitation learning to get a contract quickly and a visualization system for hands understanding to realize regular communication between players.Then,to dynamically analyze the impact of other players’unknown hands on our final reward,we design a Monte Carlo sampling algorithm based on the bidding system model(BSM)to deal with imperfect information.At the same time,a double-dummy analysis model is designed to efficiently evaluate the results of sampling.Experimental results indicate that our searching strategy outperforms the top rule-based mobile applications.展开更多
基金Supported by the National Natural Science Foundation of China under Grant Nos.52071099,52071104National Key Project of Research and Development Program under Grant No.2021YFC2801300Research Fund from National Key Laboratory of Autonomous Marine Vehicle Technology under Grant No.2023-SXJQR-SYSJJ01.
文摘The rudder mechanism of the X-rudder autonomous underwater cehicle(AUV)is relatively complex,and fault diagnosis capability is an important guarantee for its task execution in complex underwater environments.However,traditional fault diagnosis methods currently rely on prior knowledge and expert experience,and lack accuracy.In order to improve the autonomy and accuracy of fault diagnosis methods,and overcome the shortcomings of traditional algorithms,this paper proposes an X-steering AUV fault diagnosis model based on the deep reinforcement learning deep Q network(DQN)algorithm,which can learn the relationship between state data and fault types,map raw residual data to corresponding fault patterns,and achieve end-to-end mapping.In addition,to solve the problem of few X-steering fault sample data,Dropout technology is introduced during the model training phase to improve the performance of the DQN algorithm.Experimental results show that the proposed model has improved the convergence speed and comprehensive performance indicators compared to the unimproved DQN algorithm,with precision,recall,F_(1-score),and accuracy reaching up to 100%,98.07%,99.02%,and 98.50% respectively,and the model’s accuracy is higher than other machine learning algorithms like back propagation,support vector machine.
文摘Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment.Reinforcement Learning methods enable a state action function in mobile robots suited to their environment.During trial-and-error interaction with its surroundings,it helps a robot tofind an ideal behavior on its own.The Deep Q Network(DQN)algorithm is used in TurtleBot 3(TB3)to achieve the goal by successfully avoiding the obstacles.But it requires a large number of training iterations.This research mainly focuses on a mobility robot’s best path prediction utilizing DQN and the Artificial Potential Field(APF)algorithms.First,a TB3 Waffle Pi DQN is built and trained to reach the goal.Then the APF shortest path algorithm is incorporated into the DQN algorithm.The proposed planning approach is compared with the standard DQN method in a virtual environment based on the Robot Operation System(ROS).The results from the simulation show that the combination is effective for DQN and APF gives a better optimal path and takes less time when compared to the conventional DQN algo-rithm.The performance improvement rate of the proposed DQN+APF in comparison with DQN in terms of the number of successful targets is attained by 88%.The performance of the proposed DQN+APF in comparison with DQN in terms of average time is achieved by 0.331 s.The performance of the proposed DQN+APF in comparison with DQN average rewards in which the positive goal is attained by 85%and the negative goal is attained by-90%.
基金co-funded by the National Natural Science Foundation of China(No.61903187)the National Key R&D Program of China(No.2021YFB1600500)+2 种基金the China Scholarship Council(No.202006830095)the Natural Science Foundation of Jiangsu Province(No.BK20190414)the Jiangsu Province Postgraduate Innovation Fund(No.KYCX20_0213).
文摘Reinforcement Learning(RL)techniques are being studied to solve the Demand and Capacity Balancing(DCB)problems to fully exploit their computational performance.A locally gen-eralised Multi-Agent Reinforcement Learning(MARL)for real-world DCB problems is proposed.The proposed method can deploy trained agents directly to unseen scenarios in a specific Air Traffic Flow Management(ATFM)region to quickly obtain a satisfactory solution.In this method,agents of all flights in a scenario form a multi-agent decision-making system based on partial observation.The trained agent with the customised neural network can be deployed directly on the corresponding flight,allowing it to solve the DCB problem jointly.A cooperation coefficient is introduced in the reward function,which is used to adjust the agent’s cooperation preference in a multi-agent system,thereby controlling the distribution of flight delay time allocation.A multi-iteration mechanism is designed for the DCB decision-making framework to deal with problems arising from non-stationarity in MARL and to ensure that all hotspots are eliminated.Experiments based on large-scale high-complexity real-world scenarios are conducted to verify the effectiveness and efficiency of the method.From a statis-tical point of view,it is proven that the proposed method is generalised within the scope of the flights and sectors of interest,and its optimisation performance outperforms the standard computer-assisted slot allocation and state-of-the-art RL-based DCB methods.The sensitivity analysis preliminarily reveals the effect of the cooperation coefficient on delay time allocation.
基金funded by National Natural Science Foundation of China(No.62063006)Guangxi Science and Technology Major Program(No.2022AA05002)+1 种基金Key Laboratory of AI and Information Processing(Hechi University),Education Department of Guangxi Zhuang Autonomous Region(No.2022GXZDSY003)Central Leading Local Science and Technology Development Fund Project of Wuzhou(No.202201001).
文摘By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning of mobile robots.However,the traditional DDQN algorithm suffers from sparse rewards and inefficient utilization of high-quality data.Targeting those problems,an improved DDQN algorithm based on average Q-value estimation and reward redistribution was proposed.First,to enhance the precision of the target Q-value,the average of multiple previously learned Q-values from the target Q network is used to replace the single Q-value from the current target Q network.Next,a reward redistribution mechanism is designed to overcome the sparse reward problem by adjusting the final reward of each action using the round reward from trajectory information.Additionally,a reward-prioritized experience selection method is introduced,which ranks experience samples according to reward values to ensure frequent utilization of high-quality data.Finally,simulation experiments are conducted to verify the effectiveness of the proposed algorithm in fixed-position scenario and random environments.The experimental results show that compared to the traditional DDQN algorithm,the proposed algorithm achieves shorter average running time,higher average return and fewer average steps.The performance of the proposed algorithm is improved by 11.43%in the fixed scenario and 8.33%in random environments.It not only plans economic and safe paths but also significantly improves efficiency and generalization in path planning,making it suitable for widespread application in autonomous navigation and industrial automation.
基金Supported by the National Ministries and Research Funds(3020020221111)
文摘A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture adjustment. A robot is taken as an agent and trained to walk steadily on an uneven surface with obstacles, using a simple reward function based on forward progress. The reward-punishment (RP) mechanism of the DQN algorithm is established after obtaining the offline gait which was generated in advance foot trajectory planning. Instead of implementing a complex dynamic model, the proposed method enables the biped robot to learn to adjust its posture on the uneven ground and ensures walking stability. The performance and effectiveness of the proposed algorithm was validated in the V-REP simulation environment. The results demonstrate that the biped robot's lateral tile angle is less than 3° after implementing the proposed method and the walking stability is obviously improved.
基金supported by the Funds for Creative Research Groups of China under No.61921003 and Snyrey Bridge Company.
文摘Multi-agent mobile applications play an essential role in mobile applications and have attracted more and more researchers’attention.Previous work has always focused on multi-agent applications with perfect information.Researchers are usually based on human-designed rules to provide decision-making searching services.However,existing methods for solving perfect-information mobile applications cannot be directly applied to imperfect-information mobile applications.Here,we take the Contact Bridge,a multi-agent application with imperfect information,for the case study.We propose an enhanced searching strategy to deal with multi-agent applications with imperfect information.We design a self-training bidding system model and apply a Recurrent Neural Network(RNN)to model the bidding process.The bridge system model consists of two parts,a bidding prediction system based on imitation learning to get a contract quickly and a visualization system for hands understanding to realize regular communication between players.Then,to dynamically analyze the impact of other players’unknown hands on our final reward,we design a Monte Carlo sampling algorithm based on the bidding system model(BSM)to deal with imperfect information.At the same time,a double-dummy analysis model is designed to efficiently evaluate the results of sampling.Experimental results indicate that our searching strategy outperforms the top rule-based mobile applications.
