At the first sight it seems that advanced operation research is not used enough in continuous production systems as comparison with mass production, batch production and job shop systems, but really in a comprehensive...At the first sight it seems that advanced operation research is not used enough in continuous production systems as comparison with mass production, batch production and job shop systems, but really in a comprehensive evaluation the advanced operation research techniques can be used in continuous production systems in developing countries very widely, because of initial inadequate plant layout, stage by stage development of production lines, the purchase of second hand machineries from various countries, plurality of customers. A case of production system planning is proposed for a chemical company in which the above mentioned conditions are almost presented. The goals and constraints in this issue are as follows： （1） Minimizing deviation of customer＇s requirements. （2） Maximizing the profit. （3） Minimizing the frequencies of changes in formula production. （4） Minimizing the inventory of final products. （5） Balancing the production sections with regard to rate in production. （6） Limitation in inventory of raw material. The present situation is in such a way that various techniques such as goal programming, linear programming and dynamic programming can be used. But dynamic production programming issues are divided into two categories, at first one with limitation in production capacity and another with unlimited production capacity. For the first category, a systematic and acceptable solution has not been presented yet. Therefore an innovative method is used to convert the dynamic situation to a zero- one model. At last this issue is changed to a goal programming model with non-linear limitations with the use of GRG algorithm and that＇s how it is solved.展开更多
Owing to extensive applications in many fields,the synchronization problem has been widely investigated in multi-agent systems.The synchronization for multi-agent systems is a pivotal issue,which means that under the ...Owing to extensive applications in many fields,the synchronization problem has been widely investigated in multi-agent systems.The synchronization for multi-agent systems is a pivotal issue,which means that under the designed control policy,the output of systems or the state of each agent can be consistent with the leader.The purpose of this paper is to investigate a heuristic dynamic programming(HDP)-based learning tracking control for discrete-time multi-agent systems to achieve synchronization while considering disturbances in systems.Besides,due to the difficulty of solving the coupled Hamilton–Jacobi–Bellman equation analytically,an improved HDP learning control algorithm is proposed to realize the synchronization between the leader and all following agents,which is executed by an action-critic neural network.The action and critic neural network are utilized to learn the optimal control policy and cost function,respectively,by means of introducing an auxiliary action network.Finally,two numerical examples and a practical application of mobile robots are presented to demonstrate the control performance of the HDP-based learning control algorithm.展开更多
Integral reinforcement learning(IRL)is an effective tool for solving optimal control problems of nonlinear systems,and it has been widely utilized in optimal controller design for solving discrete-time nonlinearity.Ho...Integral reinforcement learning(IRL)is an effective tool for solving optimal control problems of nonlinear systems,and it has been widely utilized in optimal controller design for solving discrete-time nonlinearity.However,solving the Hamilton-Jacobi-Bellman(HJB)equations for nonlinear systems requires precise and complicated dynamics.Moreover,the research and application of IRL in continuous-time(CT)systems must be further improved.To develop the IRL of a CT nonlinear system,a data-based adaptive neural dynamic programming(ANDP)method is proposed to investigate the optimal control problem of uncertain CT multi-input systems such that the knowledge of the dynamics in the HJB equation is unnecessary.First,the multi-input model is approximated using a neural network(NN),which can be utilized to design an integral reinforcement signal.Subsequently,two criterion networks and one action network are constructed based on the integral reinforcement signal.A nonzero-sum Nash equilibrium can be reached by learning the optimal strategies of the multi-input model.In this scheme,the NN weights are constantly updated using an adaptive algorithm.The weight convergence and the system stability are analyzed in detail.The optimal control problem of a multi-input nonlinear CT system is effectively solved using the ANDP scheme,and the results are verified by a simulation study.展开更多
The residential energy scheduling of solar energy is an important research area of smart grid. On the demand side, factors such as household loads, storage batteries, the outside public utility grid and renewable ener...The residential energy scheduling of solar energy is an important research area of smart grid. On the demand side, factors such as household loads, storage batteries, the outside public utility grid and renewable energy resources, are combined together as a nonlinear, time-varying, indefinite and complex system, which is difficult to manage or optimize. Many nations have already applied the residential real-time pricing to balance the burden on their grid. In order to enhance electricity efficiency of the residential micro grid, this paper presents an action dependent heuristic dynamic programming(ADHDP) method to solve the residential energy scheduling problem. The highlights of this paper are listed below. First,the weather-type classification is adopted to establish three types of programming models based on the features of the solar energy. In addition, the priorities of different energy resources are set to reduce the loss of electrical energy transmissions.Second, three ADHDP-based neural networks, which can update themselves during applications, are designed to manage the flows of electricity. Third, simulation results show that the proposed scheduling method has effectively reduced the total electricity cost and improved load balancing process. The comparison with the particle swarm optimization algorithm further proves that the present method has a promising effect on energy management to save cost.展开更多
Control of the fed-batch ethanol fermentation processes to produce maximum product ethanol is one of the key issues in the bioreactor system.However,ethanol fermentation processes exhibit complex behavior and nonlinea...Control of the fed-batch ethanol fermentation processes to produce maximum product ethanol is one of the key issues in the bioreactor system.However,ethanol fermentation processes exhibit complex behavior and nonlinear dynamics with respect to the cell mass,substrate,feed-rate,etc.An improved dual heuristic programming algorithm based on the least squares temporal difference with gradient correction(LSTDC) algorithm(LSTDC-DHP) is proposed to solve the learning control problem of a fed-batch ethanol fermentation process.As a new algorithm of adaptive critic designs,LSTDC-DHP is used to realize online learning control of chemical dynamical plants,where LSTDC is commonly employed to approximate the value functions.Application of the LSTDC-DHP algorithm to ethanol fermentation process can realize efficient online learning control in continuous spaces.Simulation results demonstrate the effectiveness of LSTDC-DHP,and show that LSTDC-DHP can obtain the near-optimal feed rate trajectory faster than other-based algorithms.展开更多
This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge t...This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.展开更多
We address a state-of-the-art reinforcement learning(RL)control approach to automatically configure robotic pros-thesis impedance parameters to enable end-to-end,continuous locomotion intended for transfemoral amputee...We address a state-of-the-art reinforcement learning(RL)control approach to automatically configure robotic pros-thesis impedance parameters to enable end-to-end,continuous locomotion intended for transfemoral amputee subjects.Specifically,our actor-critic based RL provides tracking control of a robotic knee prosthesis to mimic the intact knee profile.This is a significant advance from our previous RL based automatic tuning of prosthesis control parameters which have centered on regulation control with a designer prescribed robotic knee profile as the target.In addition to presenting the tracking control algorithm based on direct heuristic dynamic programming(dHDP),we provide a control performance guarantee including the case of constrained inputs.We show that our proposed tracking control possesses several important properties,such as weight convergence of the learning networks,Bellman(sub)optimality of the cost-to-go value function and control input,and practical stability of the human-robot system.We further provide a systematic simulation of the proposed tracking control using a realistic human-robot system simulator,the OpenSim,to emulate how the dHDP enables level ground walking,walking on different terrains and at different paces.These results show that our proposed dHDP based tracking control is not only theoretically suitable,but also practically useful.展开更多
This paper attempts to optimize optimal capacities, block routing and mine sequencing problems in a mining system. The solution approach is based on a heuristics and the mixed integer programming (MIP). Unlike previou...This paper attempts to optimize optimal capacities, block routing and mine sequencing problems in a mining system. The solution approach is based on a heuristics and the mixed integer programming (MIP). Unlike previous sequential solution approaches, the problems are herein solved at the same time. Furthermore, the proposed approach guarantees practical solutions because it considers ore material distribution within orebody. The paper has two main contributions: (a) the proposed approach generates production rates in a manner that the capacities are satisfied; (b) the proposed approach does not use pre-defined marginal cut-off grades. Thus, idle capacity problem is eliminated and different scheduling combinations are allowed. To see the performance of the approach proposed, a case study is carried out using a gold data. The schedule generated shows that the approach can determine optimal production rates, block destination and sequencing effectively.展开更多
Intelligent confrontation has become a vital technology for future air combats.Confrontation games between a penetrating aircraft and an intercepting aircraft are essential for modern air combats.In addition,the perfo...Intelligent confrontation has become a vital technology for future air combats.Confrontation games between a penetrating aircraft and an intercepting aircraft are essential for modern air combats.In addition,the perfor-mance indexes of both the interceptor and penetrator must be considered.Traditional methods only solve one side’s guidance problem without considering the intelligence of the opponent.In this paper,an adaptive heuristic dynamic programming-based algorithm is proposed for aircraft confrontation games.This algorithm constructs a heuristic dynamic programming model for both confrontation aircraft and then updates the critical and ac-tion network parameters using the dynamic confrontation state information.Numerical simulations indicate that the proposed algorithm can optimize the guidance law for both the interceptor and penetrator and is therefore superior to traditional proportional navigation methods.展开更多
The stable multicast flow aggregation （MFA） problem in internet protocol （IP） over optical network under the dynamical scenario is studied. Given an optical network topology, there is a set of head ends and access...The stable multicast flow aggregation （MFA） problem in internet protocol （IP） over optical network under the dynamical scenario is studied. Given an optical network topology, there is a set of head ends and access touters attached to the optical network, in which each head end can provide a set of programs （IP multicasting flows） and each access router requests a set of programs, we find a set of stable light-trees to accommodate the optimally aggregated multicast IP flows if the requests of access touters changed dynamically. We introduce a program correlation matrix to describe the preference of end users＇ requests. As the original MFA problem is NP-complete, a heuristic approach, named most correlated program first （MCPF）, is presented and compared with the extended least tree first （ELTF） algorithm which is topology- aware. Simulation results show that MCPF can achieve better performance than ELTF in terms of stability with negligible increment of network resource usage.展开更多
文摘At the first sight it seems that advanced operation research is not used enough in continuous production systems as comparison with mass production, batch production and job shop systems, but really in a comprehensive evaluation the advanced operation research techniques can be used in continuous production systems in developing countries very widely, because of initial inadequate plant layout, stage by stage development of production lines, the purchase of second hand machineries from various countries, plurality of customers. A case of production system planning is proposed for a chemical company in which the above mentioned conditions are almost presented. The goals and constraints in this issue are as follows： （1） Minimizing deviation of customer＇s requirements. （2） Maximizing the profit. （3） Minimizing the frequencies of changes in formula production. （4） Minimizing the inventory of final products. （5） Balancing the production sections with regard to rate in production. （6） Limitation in inventory of raw material. The present situation is in such a way that various techniques such as goal programming, linear programming and dynamic programming can be used. But dynamic production programming issues are divided into two categories, at first one with limitation in production capacity and another with unlimited production capacity. For the first category, a systematic and acceptable solution has not been presented yet. Therefore an innovative method is used to convert the dynamic situation to a zero- one model. At last this issue is changed to a goal programming model with non-linear limitations with the use of GRG algorithm and that＇s how it is solved.
基金This work was supported by Tianjin Natural Science Foundation under Grant 20JCYBJC00880Beijing key Laboratory Open Fund of Long-Life Technology of Precise Rotation and Transmission MechanismsGuangdong Provincial Key Laboratory of Intelligent Decision and Cooperative Control.
文摘Owing to extensive applications in many fields,the synchronization problem has been widely investigated in multi-agent systems.The synchronization for multi-agent systems is a pivotal issue,which means that under the designed control policy,the output of systems or the state of each agent can be consistent with the leader.The purpose of this paper is to investigate a heuristic dynamic programming(HDP)-based learning tracking control for discrete-time multi-agent systems to achieve synchronization while considering disturbances in systems.Besides,due to the difficulty of solving the coupled Hamilton–Jacobi–Bellman equation analytically,an improved HDP learning control algorithm is proposed to realize the synchronization between the leader and all following agents,which is executed by an action-critic neural network.The action and critic neural network are utilized to learn the optimal control policy and cost function,respectively,by means of introducing an auxiliary action network.Finally,two numerical examples and a practical application of mobile robots are presented to demonstrate the control performance of the HDP-based learning control algorithm.
文摘Integral reinforcement learning(IRL)is an effective tool for solving optimal control problems of nonlinear systems,and it has been widely utilized in optimal controller design for solving discrete-time nonlinearity.However,solving the Hamilton-Jacobi-Bellman(HJB)equations for nonlinear systems requires precise and complicated dynamics.Moreover,the research and application of IRL in continuous-time(CT)systems must be further improved.To develop the IRL of a CT nonlinear system,a data-based adaptive neural dynamic programming(ANDP)method is proposed to investigate the optimal control problem of uncertain CT multi-input systems such that the knowledge of the dynamics in the HJB equation is unnecessary.First,the multi-input model is approximated using a neural network(NN),which can be utilized to design an integral reinforcement signal.Subsequently,two criterion networks and one action network are constructed based on the integral reinforcement signal.A nonzero-sum Nash equilibrium can be reached by learning the optimal strategies of the multi-input model.In this scheme,the NN weights are constantly updated using an adaptive algorithm.The weight convergence and the system stability are analyzed in detail.The optimal control problem of a multi-input nonlinear CT system is effectively solved using the ANDP scheme,and the results are verified by a simulation study.
基金supported in part by the National Natural Science Foundation of China(61533017,U1501251,61374105,61722312)
文摘The residential energy scheduling of solar energy is an important research area of smart grid. On the demand side, factors such as household loads, storage batteries, the outside public utility grid and renewable energy resources, are combined together as a nonlinear, time-varying, indefinite and complex system, which is difficult to manage or optimize. Many nations have already applied the residential real-time pricing to balance the burden on their grid. In order to enhance electricity efficiency of the residential micro grid, this paper presents an action dependent heuristic dynamic programming(ADHDP) method to solve the residential energy scheduling problem. The highlights of this paper are listed below. First,the weather-type classification is adopted to establish three types of programming models based on the features of the solar energy. In addition, the priorities of different energy resources are set to reduce the loss of electrical energy transmissions.Second, three ADHDP-based neural networks, which can update themselves during applications, are designed to manage the flows of electricity. Third, simulation results show that the proposed scheduling method has effectively reduced the total electricity cost and improved load balancing process. The comparison with the particle swarm optimization algorithm further proves that the present method has a promising effect on energy management to save cost.
基金Supported by the National Natural Science Foundation of China(61573052)
文摘Control of the fed-batch ethanol fermentation processes to produce maximum product ethanol is one of the key issues in the bioreactor system.However,ethanol fermentation processes exhibit complex behavior and nonlinear dynamics with respect to the cell mass,substrate,feed-rate,etc.An improved dual heuristic programming algorithm based on the least squares temporal difference with gradient correction(LSTDC) algorithm(LSTDC-DHP) is proposed to solve the learning control problem of a fed-batch ethanol fermentation process.As a new algorithm of adaptive critic designs,LSTDC-DHP is used to realize online learning control of chemical dynamical plants,where LSTDC is commonly employed to approximate the value functions.Application of the LSTDC-DHP algorithm to ethanol fermentation process can realize efficient online learning control in continuous spaces.Simulation results demonstrate the effectiveness of LSTDC-DHP,and show that LSTDC-DHP can obtain the near-optimal feed rate trajectory faster than other-based algorithms.
基金the National Key Research and Development Program of China(2021ZD0112302)the National Natural Science Foundation of China(62222301,61890930-5,62021003)the Beijing Natural Science Foundation(JQ19013).
文摘This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.
基金This work was partly supported by the National Science Foundation(1563921,1808752,1563454,1808898).
文摘We address a state-of-the-art reinforcement learning(RL)control approach to automatically configure robotic pros-thesis impedance parameters to enable end-to-end,continuous locomotion intended for transfemoral amputee subjects.Specifically,our actor-critic based RL provides tracking control of a robotic knee prosthesis to mimic the intact knee profile.This is a significant advance from our previous RL based automatic tuning of prosthesis control parameters which have centered on regulation control with a designer prescribed robotic knee profile as the target.In addition to presenting the tracking control algorithm based on direct heuristic dynamic programming(dHDP),we provide a control performance guarantee including the case of constrained inputs.We show that our proposed tracking control possesses several important properties,such as weight convergence of the learning networks,Bellman(sub)optimality of the cost-to-go value function and control input,and practical stability of the human-robot system.We further provide a systematic simulation of the proposed tracking control using a realistic human-robot system simulator,the OpenSim,to emulate how the dHDP enables level ground walking,walking on different terrains and at different paces.These results show that our proposed dHDP based tracking control is not only theoretically suitable,but also practically useful.
文摘This paper attempts to optimize optimal capacities, block routing and mine sequencing problems in a mining system. The solution approach is based on a heuristics and the mixed integer programming (MIP). Unlike previous sequential solution approaches, the problems are herein solved at the same time. Furthermore, the proposed approach guarantees practical solutions because it considers ore material distribution within orebody. The paper has two main contributions: (a) the proposed approach generates production rates in a manner that the capacities are satisfied; (b) the proposed approach does not use pre-defined marginal cut-off grades. Thus, idle capacity problem is eliminated and different scheduling combinations are allowed. To see the performance of the approach proposed, a case study is carried out using a gold data. The schedule generated shows that the approach can determine optimal production rates, block destination and sequencing effectively.
基金supported by the China Postdoctoral Science Founda-tion(Grant No.2020M681750).
文摘Intelligent confrontation has become a vital technology for future air combats.Confrontation games between a penetrating aircraft and an intercepting aircraft are essential for modern air combats.In addition,the perfor-mance indexes of both the interceptor and penetrator must be considered.Traditional methods only solve one side’s guidance problem without considering the intelligence of the opponent.In this paper,an adaptive heuristic dynamic programming-based algorithm is proposed for aircraft confrontation games.This algorithm constructs a heuristic dynamic programming model for both confrontation aircraft and then updates the critical and ac-tion network parameters using the dynamic confrontation state information.Numerical simulations indicate that the proposed algorithm can optimize the guidance law for both the interceptor and penetrator and is therefore superior to traditional proportional navigation methods.
基金the National Natural Science Foundation of China (No.60502004) and the"863"Program of China.
文摘The stable multicast flow aggregation （MFA） problem in internet protocol （IP） over optical network under the dynamical scenario is studied. Given an optical network topology, there is a set of head ends and access touters attached to the optical network, in which each head end can provide a set of programs （IP multicasting flows） and each access router requests a set of programs, we find a set of stable light-trees to accommodate the optimally aggregated multicast IP flows if the requests of access touters changed dynamically. We introduce a program correlation matrix to describe the preference of end users＇ requests. As the original MFA problem is NP-complete, a heuristic approach, named most correlated program first （MCPF）, is presented and compared with the extended least tree first （ELTF） algorithm which is topology- aware. Simulation results show that MCPF can achieve better performance than ELTF in terms of stability with negligible increment of network resource usage.
