Multi-agent reinforcement learning(MARL)has proven its effectiveness in cooperative multi-agent systems(MASs)but still faces issues on the curse of dimensionality and learning efficiency.The main difficulty is caused ...Multi-agent reinforcement learning(MARL)has proven its effectiveness in cooperative multi-agent systems(MASs)but still faces issues on the curse of dimensionality and learning efficiency.The main difficulty is caused by the strong inter-agent coupling nature embedded in an MARL problem,which is yet to be fully exploited in existing algorithms.In this work,we recognize a learning graph characterizing the dependence between individual rewards and individual policies.Then we propose a graph-based reward aggregation(GRA)method,which utilizes the inherent coupling relationship among agents to eliminate redundant information.Specifically,GRA passes information among cooperating agents through graph attention networks to obtain aggregated rewards that contribute to the fitting of the value function,making each agent learn a decentralized executable cooperation policy.In addition,we propose a variant of GRA,named GRA-decen,which achieves decentralized training and decentralized execution(DTDE)when each agent only has access to information of partial agents in the learning process.We conduct experiments in different environments and demonstrate the practicality and scalability of our algorithms.展开更多
With the advent of sixth-generation mobile communications(6G),space-air-ground integrated networks have become mainstream.This paper focuses on collaborative scheduling for mobile edge computing(MEC)under a three-tier...With the advent of sixth-generation mobile communications(6G),space-air-ground integrated networks have become mainstream.This paper focuses on collaborative scheduling for mobile edge computing(MEC)under a three-tier heterogeneous architecture composed of mobile devices,unmanned aerial vehicles(UAVs),and macro base stations(BSs).This scenario typically faces fast channel fading,dynamic computational loads,and energy constraints,whereas classical queuing-theoretic or convex-optimization approaches struggle to yield robust solutions in highly dynamic settings.To address this issue,we formulate a multi-agent Markov decision process(MDP)for an air-ground-fused MEC system,unify link selection,bandwidth/power allocation,and task offloading into a continuous action space and propose a joint scheduling strategy that is based on an improved MATD3 algorithm.The improvements include Alternating Layer Normalization(ALN)in the actor to suppress gradient variance,Residual Orthogonalization(RO)in the critic to reduce the correlation between the twin Q-value estimates,and a dynamic-temperature reward to enable adaptive trade-offs during training.On a multi-user,dual-link simulation platform,we conduct ablation and baseline comparisons.The results reveal that the proposed method has better convergence and stability.Compared with MADDPG,TD3,and DSAC,our algorithm achieves more robust performance across key metrics.展开更多
This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary obj...This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary objective is to explore the unknown environments to locate and track targets effectively. To address this problem, we propose a novel Multi-Agent Reinforcement Learning (MARL) method based on Graph Neural Network (GNN). Firstly, a method is introduced for encoding continuous-space multi-UAV problem data into spatial graphs which establish essential relationships among agents, obstacles, and targets. Secondly, a Graph AttenTion network (GAT) model is presented, which focuses exclusively on adjacent nodes, learns attention weights adaptively and allows agents to better process information in dynamic environments. Reward functions are specifically designed to tackle exploration challenges in environments with sparse rewards. By introducing a framework that integrates centralized training and distributed execution, the advancement of models is facilitated. Simulation results show that the proposed method outperforms the existing MARL method in search rate and tracking performance with less collisions. The experiments show that the proposed method can be extended to applications with a larger number of agents, which provides a potential solution to the challenging problem of multi-UAV autonomous tracking in dynamic unknown environments.展开更多
The increasing adoption of renewable energy has posed challenges for voltage regulation in power distribution networks.Gridaware energy management,which includes the control of smart inverters and energy management sy...The increasing adoption of renewable energy has posed challenges for voltage regulation in power distribution networks.Gridaware energy management,which includes the control of smart inverters and energy management systems,is a trending way to mitigate this problem.However,existing multi-agent reinforcement learning methods for grid-aware energy management have not sufficiently considered the importance of agent cooperation and the unique characteristics of the grid,which leads to limited performance.In this study,we propose a new approach named multi-agent hierarchical graph attention reinforcement learning framework(MAHGA)to stabilize the voltage.Specifically,under the paradigm of centralized training and decentralized execution,we model the power distribution network as a novel hierarchical graph containing the agent-level topology and the bus-level topology.Then a hierarchical graph attention model is devised to capture the complex correlation between agents.Moreover,we incorporate graph contrastive learning as an auxiliary task in the reinforcement learning process to improve representation learning from graphs.Experiments on several real-world scenarios reveal that our approach achieves the best performance and can reduce the number of voltage violations remarkably.展开更多
The application of multiple unmanned aerial vehicles(UAVs)for the pursuit and capture of unauthorized UAVs has emerged as a novel approach to ensuring the safety of urban airspace.However,pursuit UAVs necessitate the ...The application of multiple unmanned aerial vehicles(UAVs)for the pursuit and capture of unauthorized UAVs has emerged as a novel approach to ensuring the safety of urban airspace.However,pursuit UAVs necessitate the utilization of their own sensors to proactively gather information from the unauthorized UAV.Considering the restricted sensing range of sensors,this paper proposes a multi-UAV with limited visual field pursuit-evasion(MUV-PE)problem.Each pursuer has a visual field characterized by limited perception distance and viewing angle,potentially obstructed by buildings.Only when the unauthorized UAV,i.e.,the evader,enters the visual field of any pursuer can its position be acquired.The objective of the pursuers is to capture the evader as soon as possible without collision.To address this problem,we propose the normalizing flow actor with graph attention critic(NAGC)algorithm,a multi-agent reinforcement learning(MARL)approach.NAGC executes normalizing flows to augment the flexibility of policy network,enabling the agent to sample actions from more intricate distributions rather than common distributions.To enhance the capability of simultaneously comprehending spatial relationships among multiple UAVs and environmental obstacles,NAGC integrates the“obstacle-target”graph attention networks,significantly aiding pursuers in supporting search or pursuit activities.Extensive experiments conducted in a high-precision simulator validate the promising performance of the NAGC algorithm.展开更多
Container-based virtualization technology has been more widely used in edge computing environments recently due to its advantages of lighter resource occupation, faster startup capability, and better resource utilizat...Container-based virtualization technology has been more widely used in edge computing environments recently due to its advantages of lighter resource occupation, faster startup capability, and better resource utilization efficiency. To meet the diverse needs of tasks, it usually needs to instantiate multiple network functions in the form of containers interconnect various generated containers to build a Container Cluster(CC). Then CCs will be deployed on edge service nodes with relatively limited resources. However, the increasingly complex and timevarying nature of tasks brings great challenges to optimal placement of CC. This paper regards the charges for various resources occupied by providing services as revenue, the service efficiency and energy consumption as cost, thus formulates a Mixed Integer Programming(MIP) model to describe the optimal placement of CC on edge service nodes. Furthermore, an Actor-Critic based Deep Reinforcement Learning(DRL) incorporating Graph Convolutional Networks(GCN) framework named as RL-GCN is proposed to solve the optimization problem. The framework obtains an optimal placement strategy through self-learning according to the requirements and objectives of the placement of CC. Particularly, through the introduction of GCN, the features of the association relationship between multiple containers in CCs can be effectively extracted to improve the quality of placement.The experiment results show that under different scales of service nodes and task requests, the proposed method can obtain the improved system performance in terms of placement error ratio, time efficiency of solution output and cumulative system revenue compared with other representative baseline methods.展开更多
To guarantee the heterogeneous delay requirements of the diverse vehicular services,it is necessary to design a full cooperative policy for both Vehicle to Infrastructure(V2I)and Vehicle to Vehicle(V2V)links.This pape...To guarantee the heterogeneous delay requirements of the diverse vehicular services,it is necessary to design a full cooperative policy for both Vehicle to Infrastructure(V2I)and Vehicle to Vehicle(V2V)links.This paper investigates the reduction of the delay in edge information sharing for V2V links while satisfying the delay requirements of the V2I links.Specifically,a mean delay minimization problem and a maximum individual delay minimization problem are formulated to improve the global network performance and ensure the fairness of a single user,respectively.A multi-agent reinforcement learning framework is designed to solve these two problems,where a new reward function is proposed to evaluate the utilities of the two optimization objectives in a unified framework.Thereafter,a proximal policy optimization approach is proposed to enable each V2V user to learn its policy using the shared global network reward.The effectiveness of the proposed approach is finally validated by comparing the obtained results with those of the other baseline approaches through extensive simulation experiments.展开更多
Reinforcement Learning(RL)techniques are being studied to solve the Demand and Capacity Balancing(DCB)problems to fully exploit their computational performance.A locally gen-eralised Multi-Agent Reinforcement Learning...Reinforcement Learning(RL)techniques are being studied to solve the Demand and Capacity Balancing(DCB)problems to fully exploit their computational performance.A locally gen-eralised Multi-Agent Reinforcement Learning(MARL)for real-world DCB problems is proposed.The proposed method can deploy trained agents directly to unseen scenarios in a specific Air Traffic Flow Management(ATFM)region to quickly obtain a satisfactory solution.In this method,agents of all flights in a scenario form a multi-agent decision-making system based on partial observation.The trained agent with the customised neural network can be deployed directly on the corresponding flight,allowing it to solve the DCB problem jointly.A cooperation coefficient is introduced in the reward function,which is used to adjust the agent’s cooperation preference in a multi-agent system,thereby controlling the distribution of flight delay time allocation.A multi-iteration mechanism is designed for the DCB decision-making framework to deal with problems arising from non-stationarity in MARL and to ensure that all hotspots are eliminated.Experiments based on large-scale high-complexity real-world scenarios are conducted to verify the effectiveness and efficiency of the method.From a statis-tical point of view,it is proven that the proposed method is generalised within the scope of the flights and sectors of interest,and its optimisation performance outperforms the standard computer-assisted slot allocation and state-of-the-art RL-based DCB methods.The sensitivity analysis preliminarily reveals the effect of the cooperation coefficient on delay time allocation.展开更多
Vehicular edge computing(VEC)is emerging as a promising solution paradigm to meet the requirements of compute-intensive applications in internet of vehicle(IoV).Non-orthogonal multiple access(NOMA)has advantages in im...Vehicular edge computing(VEC)is emerging as a promising solution paradigm to meet the requirements of compute-intensive applications in internet of vehicle(IoV).Non-orthogonal multiple access(NOMA)has advantages in improving spectrum efficiency and dealing with bandwidth scarcity and cost.It is an encouraging progress combining VEC and NOMA.In this paper,we jointly optimize task offloading decision and resource allocation to maximize the service utility of the NOMA-VEC system.To solve the optimization problem,we propose a multiagent deep graph reinforcement learning algorithm.The algorithm extracts the topological features and relationship information between agents from the system state as observations,outputs task offloading decision and resource allocation simultaneously with local policy network,which is updated by a local learner.Simulation results demonstrate that the proposed method achieves a 1.52%∼5.80%improvement compared with the benchmark algorithms in system service utility.展开更多
Recently,Network Functions Virtualization(NFV)has become a critical resource for optimizing capability utilization in the 5G/B5G era.NFV decomposes the network resource paradigm,demonstrating the efficient utilization...Recently,Network Functions Virtualization(NFV)has become a critical resource for optimizing capability utilization in the 5G/B5G era.NFV decomposes the network resource paradigm,demonstrating the efficient utilization of Network Functions(NFs)to enable configurable service priorities and resource demands.Telecommunications Service Providers(TSPs)face challenges in network utilization,as the vast amounts of data generated by the Internet of Things(IoT)overwhelm existing infrastructures.IoT applications,which generate massive volumes of diverse data and require real-time communication,contribute to bottlenecks and congestion.In this context,Multiaccess Edge Computing(MEC)is employed to support resource and priority-aware IoT applications by implementing Virtual Network Function(VNF)sequences within Service Function Chaining(SFC).This paper proposes the use of Deep Reinforcement Learning(DRL)combined with Graph Neural Networks(GNN)to enhance network processing,performance,and resource pooling capabilities.GNN facilitates feature extraction through Message-Passing Neural Network(MPNN)mechanisms.Together with DRL,Deep Q-Networks(DQN)are utilized to dynamically allocate resources based on IoT network priorities and demands.Our focus is on minimizing delay times for VNF instance execution,ensuring effective resource placement,and allocation in SFC deployments,offering flexibility to adapt to real-time changes in priority and workload.Simulation results demonstrate that our proposed scheme outperforms reference models in terms of reward,delay,delivery,service drop ratios,and average completion ratios,proving its potential for IoT applications.展开更多
This article studies the effective traffic signal control problem of multiple intersections in a city-level traffic system.A novel regional multi-agent cooperative reinforcement learning algorithm called RegionSTLight...This article studies the effective traffic signal control problem of multiple intersections in a city-level traffic system.A novel regional multi-agent cooperative reinforcement learning algorithm called RegionSTLight is proposed to improve the traffic efficiency.Firstly a regional multi-agent Q-learning framework is proposed,which can equivalently decompose the global Q value of the traffic system into the local values of several regions Based on the framework and the idea of human-machine cooperation,a dynamic zoning method is designed to divide the traffic network into several strong-coupled regions according to realtime traffic flow densities.In order to achieve better cooperation inside each region,a lightweight spatio-temporal fusion feature extraction network is designed.The experiments in synthetic real-world and city-level scenarios show that the proposed RegionS TLight converges more quickly,is more stable,and obtains better asymptotic performance compared to state-of-theart models.展开更多
The majority of existing graph-network-based few-shot models focus on a node-similarity update mode.The lack of adequate information intensies the risk of overtraining.In this paper,we propose a novel Multihead Attent...The majority of existing graph-network-based few-shot models focus on a node-similarity update mode.The lack of adequate information intensies the risk of overtraining.In this paper,we propose a novel Multihead Attention Graph Network to excavate discriminative relation and fulll effective information propagation.For edge update,the node-level attention is used to evaluate the similarities between the two nodes and the distributionlevel attention extracts more in-deep global relation.The cooperation between those two parts provides a discriminative and comprehensive expression for edge feature.For node update,we embrace the label-level attention to soften the noise of irrelevant nodes and optimize the update direction.Our proposed model is veried through extensive experiments on two few-shot benchmark MiniImageNet and CIFAR-FS dataset.The results suggest that our method has a strong capability of noise immunity and quick convergence.The classication accuracy outperforms most state-of-the-art approaches.展开更多
Aiming at the rapid growth of network services,which leads to the problems of long service request processing time and high deployment cost in the deployment of network function virtualization service function chain(S...Aiming at the rapid growth of network services,which leads to the problems of long service request processing time and high deployment cost in the deployment of network function virtualization service function chain(SFC)under 5G networks,this paper proposes a multi-agent deep deterministic policy gradient optimization algorithm for SFC deployment(MADDPG-SD).Initially,an optimization model is devised to enhance the request acceptance rate,minimizing the latency and deploying the cost SFC is constructed for the network resource-constrained case.Subsequently,we model the dynamic problem as a Markov decision process(MDP),facilitating adaptation to the evolving states of network resources.Finally,by allocating SFCs to different agents and adopting a collaborative deployment strategy,each agent aims to maximize the request acceptance rate or minimize latency and costs.These agents learn strategies from historical data of virtual network functions in SFCs to guide server node selection,and achieve approximately optimal SFC deployment strategies through a cooperative framework of centralized training and distributed execution.Experimental simulation results indicate that the proposed method,while simultaneously meeting performance requirements and resource capacity constraints,has effectively increased the acceptance rate of requests compared to the comparative algorithms,reducing the end-to-end latency by 4.942%and the deployment cost by 8.045%.展开更多
This paper focuses on the problem of multi-station multi-robot spot welding task assignment,and proposes a deep reinforcement learning(DRL)framework,which is made up of a public graph attention network and independent...This paper focuses on the problem of multi-station multi-robot spot welding task assignment,and proposes a deep reinforcement learning(DRL)framework,which is made up of a public graph attention network and independent policy networks.The graph of welding spots distribution is encoded using the graph attention network.Independent policy networks with attention mechanism as a decoder can handle the encoded graph and decide to assign robots to different tasks.The policy network is used to convert the large scale welding spots allocation problem to multiple small scale singlerobot welding path planning problems,and the path planning problem is quickly solved through existing methods.Then,the model is trained through reinforcement learning.In addition,the task balancing method is used to allocate tasks to multiple stations.The proposed algorithm is compared with classical algorithms,and the results show that the algorithm based on DRL can produce higher quality solutions.展开更多
Anomaly fluctuations in operating conditions, catalyst wear, crushing, and the deterioration of feedstock properties in fluid catalytic cracking (FCC) units can disrupt the normal circulating fluidization process of t...Anomaly fluctuations in operating conditions, catalyst wear, crushing, and the deterioration of feedstock properties in fluid catalytic cracking (FCC) units can disrupt the normal circulating fluidization process of the catalyst. Although several effective models have been proposed in previous research to address anomaly detection in chemical processes, most fail to adequately capture the spatial-temporal dependencies of multi-source, mixed-frequency information. In this study, an innovative multi-source mixed-frequency information fusion framework based on a spatial-temporal graph attention network (MIF-STGAT) is proposed to investigate the causes of FCC regenerator catalyst loss anomalies for guide onsite operational management, enhancing the long-term stability of FCC unit operations. First, a reconstruction-based dual-encoder-decoder framework is developed to facilitate the acquisition of mixed-frequency features and information fusion during the FCC regenerator catalyst loss process. Subsequently, a graph attention network and a multilayer long short-term memory network with a differential structure are integrated into the reconstruction-based dual-encoder-shared-decoder framework to capture the dynamic fluctuations and critical features associated with anomalies. Experimental results from the Chinese FCC industrial process demonstrate that MIF-STGAT achieves excellent accuracy and interpretability for anomaly detection.展开更多
It is known that long non-coding RNAs(lncRNAs)play vital roles in biological processes and contribute to the progression,development,and treatment of various diseases.Obviously,understanding associations between disea...It is known that long non-coding RNAs(lncRNAs)play vital roles in biological processes and contribute to the progression,development,and treatment of various diseases.Obviously,understanding associations between diseases and lncRNAs significantly enhances our ability to interpret disease mechanisms.Nevertheless,the process of determining lncRNA-disease associations is costly,labor-intensive,and time-consuming.Hence,it is expected to foster computational strategies to uncover lncRNA-disease relationships for further verification to save time and resources.In this study,a collaborative filtering and graph attention network-based LncRNA-Disease Association(CFGANLDA)method was nominated to expose potential lncRNA-disease associations.First,it takes into account the advantages of using biological information from multiple sources.Next,it uses a collaborative filtering technique in order to address the sparse data problem.It also employs a graph attention network to reinforce both linear and non-linear features of the associations to advance prediction performance.The computational results indicate that CFGANLDA gains better prediction performance compared to other state-of-the-art approaches.The CFGANLDA’s area under the receiver operating characteristic curve(AUC)metric is 0.9835,whereas its area under the precision-recall curve(AUPR)metric is 0.9822.Statistical analysis using 10-fold cross-validation experiments proves that these metrics are significant.Furthermore,three case studies on prostate,liver,and stomach cancers attest to the validity of CFGANLDA performance.As a result,CFGANLDA method proves to be a valued tool for lncRNA-disease association prediction.展开更多
In the domain of knowledge graph embedding,conventional approaches typically transform entities and relations into continuous vector spaces.However,parameter efficiency becomes increasingly crucial when dealing with l...In the domain of knowledge graph embedding,conventional approaches typically transform entities and relations into continuous vector spaces.However,parameter efficiency becomes increasingly crucial when dealing with large-scale knowledge graphs that contain vast numbers of entities and relations.In particular,resource-intensive embeddings often lead to increased computational costs,and may limit scalability and adaptability in practical environ-ments,such as in low-resource settings or real-world applications.This paper explores an approach to knowledge graph representation learning that leverages small,reserved entities and relation sets for parameter-efficient embedding.We introduce a hierarchical attention network designed to refine and maximize the representational quality of embeddings by selectively focusing on these reserved sets,thereby reducing model complexity.Empirical assessments validate that our model achieves high performance on the benchmark dataset with fewer parameters and smaller embedding dimensions.The ablation studies further highlight the impact and contribution of each component in the proposed hierarchical attention structure.展开更多
Industrial Internet combines the industrial system with Internet connectivity to build a new manufacturing and service system covering the entire industry chain and value chain.Its highly heterogeneous network structu...Industrial Internet combines the industrial system with Internet connectivity to build a new manufacturing and service system covering the entire industry chain and value chain.Its highly heterogeneous network structure and diversified application requirements call for the applying of network slicing technology.Guaranteeing robust network slicing is essential for Industrial Internet,but it faces the challenge of complex slice topologies caused by the intricate interaction relationships among Network Functions(NFs)composing the slice.Existing works have not concerned the strengthening problem of industrial network slicing regarding its complex network properties.Towards this end,we aim to study this issue by intelligently selecting a subset of most valuable NFs with the minimum cost to satisfy the strengthening requirements.State-of-the-art AlphaGo series of algorithms and the advanced graph neural network technology are combined to build the solution.Simulation results demonstrate the superior performance of our scheme compared to the benchmark schemes.展开更多
The proliferation of distributed energy resources and time-varying network topologies in active distribution networks presents unprecedented challenges for network operators.While reinforcement learning (RL) has shown...The proliferation of distributed energy resources and time-varying network topologies in active distribution networks presents unprecedented challenges for network operators.While reinforcement learning (RL) has shown promise in addressing network-constrained energy scheduling,it faces difficulties in managing the complexities of dynamic topologies and discrete-continuous hybrid action spaces.To address these challenges,a graph-based safe RL approach is proposed to learn dynamic optimal power flow under time-varying network topologies.This proposed approach leverages graph convolution operators to handle network topology changes,while safe RL with parameterized action ensures policy development.Specifically,the graph convolution operator abstracts key characteristics of the network topology,enabling effective power flow management in non-stationary environments.Besides that,a parameterized action constrained Markov decision process is employed to handle the hybrid action space and ensure compliance with physical network constraints,thereby accelerating the deployment of safe policy for hybrid action spaces.Numerical results demonstrate that the proposed approach efficiently navigates the discrete-continuous decision space while accounting for the constraints imposed by the dynamic nature of power flow in time-varying network topologies.展开更多
The high proportion of renewable energy integration and the dynamic changes in grid topology necessitate the enhancement of voltage/var control(VVC)to manage voltage fluctuations more rapidly.Traditional model-based c...The high proportion of renewable energy integration and the dynamic changes in grid topology necessitate the enhancement of voltage/var control(VVC)to manage voltage fluctuations more rapidly.Traditional model-based control algorithms are becoming increasingly incompetent for VVC due to their high model dependence and slow online computation speed.To alleviate these issues,this paper introduces a graph attention network(GAT)based deep reinforcement learning for VVC of topologically variable power system.Firstly,combining the physical information of the actual power grid,a physics-informed GAT is proposed and embedded into the proximal policy optimization(PPO)algorithm.The GAT-PPO algorithm can capture topological and spatial correlations among the node features to tackle topology changes.To address the slow training,the Relief F-S algorithm identifies critical state variables,significantly reducing the dimensionality of state space.Then,the training samples retained in the experience buffer are designed to mitigate the sparse reward issue.Finally,the validation on the modified IEEE 39-bus system and an actual power grid demonstrates superior performance of the proposed algorithm compared with state-of-the-art algorithms,including PPO algorithm and twin delayed deep deterministic policy gradient(TD3)algorithm.The proposed algorithm exhibits enhanced convergence during training,faster solution speed,and improved VVC performance,even in scenarios involving grid topology changes and increased renewable energy integration.Meanwhile,in the adopted cases,the network loss is reduced by 6.9%,10.8%,and 7.7%,respectively,demonstrating favorable economic outcomes.展开更多
基金supported in part by the National Natural Science Foundation of China(grants 62203073 and 62573068)the Natural Science Foundation of Chongqing,China(grant CSTB2022NSCQMSX0577)。
文摘Multi-agent reinforcement learning(MARL)has proven its effectiveness in cooperative multi-agent systems(MASs)but still faces issues on the curse of dimensionality and learning efficiency.The main difficulty is caused by the strong inter-agent coupling nature embedded in an MARL problem,which is yet to be fully exploited in existing algorithms.In this work,we recognize a learning graph characterizing the dependence between individual rewards and individual policies.Then we propose a graph-based reward aggregation(GRA)method,which utilizes the inherent coupling relationship among agents to eliminate redundant information.Specifically,GRA passes information among cooperating agents through graph attention networks to obtain aggregated rewards that contribute to the fitting of the value function,making each agent learn a decentralized executable cooperation policy.In addition,we propose a variant of GRA,named GRA-decen,which achieves decentralized training and decentralized execution(DTDE)when each agent only has access to information of partial agents in the learning process.We conduct experiments in different environments and demonstrate the practicality and scalability of our algorithms.
文摘With the advent of sixth-generation mobile communications(6G),space-air-ground integrated networks have become mainstream.This paper focuses on collaborative scheduling for mobile edge computing(MEC)under a three-tier heterogeneous architecture composed of mobile devices,unmanned aerial vehicles(UAVs),and macro base stations(BSs).This scenario typically faces fast channel fading,dynamic computational loads,and energy constraints,whereas classical queuing-theoretic or convex-optimization approaches struggle to yield robust solutions in highly dynamic settings.To address this issue,we formulate a multi-agent Markov decision process(MDP)for an air-ground-fused MEC system,unify link selection,bandwidth/power allocation,and task offloading into a continuous action space and propose a joint scheduling strategy that is based on an improved MATD3 algorithm.The improvements include Alternating Layer Normalization(ALN)in the actor to suppress gradient variance,Residual Orthogonalization(RO)in the critic to reduce the correlation between the twin Q-value estimates,and a dynamic-temperature reward to enable adaptive trade-offs during training.On a multi-user,dual-link simulation platform,we conduct ablation and baseline comparisons.The results reveal that the proposed method has better convergence and stability.Compared with MADDPG,TD3,and DSAC,our algorithm achieves more robust performance across key metrics.
基金supported by the National Natural Science Foundation of China(Nos.12272104,U22B2013).
文摘This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary objective is to explore the unknown environments to locate and track targets effectively. To address this problem, we propose a novel Multi-Agent Reinforcement Learning (MARL) method based on Graph Neural Network (GNN). Firstly, a method is introduced for encoding continuous-space multi-UAV problem data into spatial graphs which establish essential relationships among agents, obstacles, and targets. Secondly, a Graph AttenTion network (GAT) model is presented, which focuses exclusively on adjacent nodes, learns attention weights adaptively and allows agents to better process information in dynamic environments. Reward functions are specifically designed to tackle exploration challenges in environments with sparse rewards. By introducing a framework that integrates centralized training and distributed execution, the advancement of models is facilitated. Simulation results show that the proposed method outperforms the existing MARL method in search rate and tracking performance with less collisions. The experiments show that the proposed method can be extended to applications with a larger number of agents, which provides a potential solution to the challenging problem of multi-UAV autonomous tracking in dynamic unknown environments.
基金supported by National Key R&D Program of China under Grant No.2022ZD0119802National Natural Science Foundation of China under Grant No.61836011.
文摘The increasing adoption of renewable energy has posed challenges for voltage regulation in power distribution networks.Gridaware energy management,which includes the control of smart inverters and energy management systems,is a trending way to mitigate this problem.However,existing multi-agent reinforcement learning methods for grid-aware energy management have not sufficiently considered the importance of agent cooperation and the unique characteristics of the grid,which leads to limited performance.In this study,we propose a new approach named multi-agent hierarchical graph attention reinforcement learning framework(MAHGA)to stabilize the voltage.Specifically,under the paradigm of centralized training and decentralized execution,we model the power distribution network as a novel hierarchical graph containing the agent-level topology and the bus-level topology.Then a hierarchical graph attention model is devised to capture the complex correlation between agents.Moreover,we incorporate graph contrastive learning as an auxiliary task in the reinforcement learning process to improve representation learning from graphs.Experiments on several real-world scenarios reveal that our approach achieves the best performance and can reduce the number of voltage violations remarkably.
基金supported in part by the National Natural Science Foundation of China(62373380)。
文摘The application of multiple unmanned aerial vehicles(UAVs)for the pursuit and capture of unauthorized UAVs has emerged as a novel approach to ensuring the safety of urban airspace.However,pursuit UAVs necessitate the utilization of their own sensors to proactively gather information from the unauthorized UAV.Considering the restricted sensing range of sensors,this paper proposes a multi-UAV with limited visual field pursuit-evasion(MUV-PE)problem.Each pursuer has a visual field characterized by limited perception distance and viewing angle,potentially obstructed by buildings.Only when the unauthorized UAV,i.e.,the evader,enters the visual field of any pursuer can its position be acquired.The objective of the pursuers is to capture the evader as soon as possible without collision.To address this problem,we propose the normalizing flow actor with graph attention critic(NAGC)algorithm,a multi-agent reinforcement learning(MARL)approach.NAGC executes normalizing flows to augment the flexibility of policy network,enabling the agent to sample actions from more intricate distributions rather than common distributions.To enhance the capability of simultaneously comprehending spatial relationships among multiple UAVs and environmental obstacles,NAGC integrates the“obstacle-target”graph attention networks,significantly aiding pursuers in supporting search or pursuit activities.Extensive experiments conducted in a high-precision simulator validate the promising performance of the NAGC algorithm.
文摘Container-based virtualization technology has been more widely used in edge computing environments recently due to its advantages of lighter resource occupation, faster startup capability, and better resource utilization efficiency. To meet the diverse needs of tasks, it usually needs to instantiate multiple network functions in the form of containers interconnect various generated containers to build a Container Cluster(CC). Then CCs will be deployed on edge service nodes with relatively limited resources. However, the increasingly complex and timevarying nature of tasks brings great challenges to optimal placement of CC. This paper regards the charges for various resources occupied by providing services as revenue, the service efficiency and energy consumption as cost, thus formulates a Mixed Integer Programming(MIP) model to describe the optimal placement of CC on edge service nodes. Furthermore, an Actor-Critic based Deep Reinforcement Learning(DRL) incorporating Graph Convolutional Networks(GCN) framework named as RL-GCN is proposed to solve the optimization problem. The framework obtains an optimal placement strategy through self-learning according to the requirements and objectives of the placement of CC. Particularly, through the introduction of GCN, the features of the association relationship between multiple containers in CCs can be effectively extracted to improve the quality of placement.The experiment results show that under different scales of service nodes and task requests, the proposed method can obtain the improved system performance in terms of placement error ratio, time efficiency of solution output and cumulative system revenue compared with other representative baseline methods.
基金supported in part by the National Natural Science Foundation of China under grants 61901078,61771082,61871062,and U20A20157in part by the Science and Technology Research Program of Chongqing Municipal Education Commission under grant KJQN201900609+2 种基金in part by the Natural Science Foundation of Chongqing under grant cstc2020jcyj-zdxmX0024in part by University Innovation Research Group of Chongqing under grant CXQT20017in part by the China University Industry-University-Research Collaborative Innovation Fund(Future Network Innovation Research and Application Project)under grant 2021FNA04008.
文摘To guarantee the heterogeneous delay requirements of the diverse vehicular services,it is necessary to design a full cooperative policy for both Vehicle to Infrastructure(V2I)and Vehicle to Vehicle(V2V)links.This paper investigates the reduction of the delay in edge information sharing for V2V links while satisfying the delay requirements of the V2I links.Specifically,a mean delay minimization problem and a maximum individual delay minimization problem are formulated to improve the global network performance and ensure the fairness of a single user,respectively.A multi-agent reinforcement learning framework is designed to solve these two problems,where a new reward function is proposed to evaluate the utilities of the two optimization objectives in a unified framework.Thereafter,a proximal policy optimization approach is proposed to enable each V2V user to learn its policy using the shared global network reward.The effectiveness of the proposed approach is finally validated by comparing the obtained results with those of the other baseline approaches through extensive simulation experiments.
基金co-funded by the National Natural Science Foundation of China(No.61903187)the National Key R&D Program of China(No.2021YFB1600500)+2 种基金the China Scholarship Council(No.202006830095)the Natural Science Foundation of Jiangsu Province(No.BK20190414)the Jiangsu Province Postgraduate Innovation Fund(No.KYCX20_0213).
文摘Reinforcement Learning(RL)techniques are being studied to solve the Demand and Capacity Balancing(DCB)problems to fully exploit their computational performance.A locally gen-eralised Multi-Agent Reinforcement Learning(MARL)for real-world DCB problems is proposed.The proposed method can deploy trained agents directly to unseen scenarios in a specific Air Traffic Flow Management(ATFM)region to quickly obtain a satisfactory solution.In this method,agents of all flights in a scenario form a multi-agent decision-making system based on partial observation.The trained agent with the customised neural network can be deployed directly on the corresponding flight,allowing it to solve the DCB problem jointly.A cooperation coefficient is introduced in the reward function,which is used to adjust the agent’s cooperation preference in a multi-agent system,thereby controlling the distribution of flight delay time allocation.A multi-iteration mechanism is designed for the DCB decision-making framework to deal with problems arising from non-stationarity in MARL and to ensure that all hotspots are eliminated.Experiments based on large-scale high-complexity real-world scenarios are conducted to verify the effectiveness and efficiency of the method.From a statis-tical point of view,it is proven that the proposed method is generalised within the scope of the flights and sectors of interest,and its optimisation performance outperforms the standard computer-assisted slot allocation and state-of-the-art RL-based DCB methods.The sensitivity analysis preliminarily reveals the effect of the cooperation coefficient on delay time allocation.
基金supported by the Talent Fund of Beijing Jiaotong University(No.2023XKRC028)CCFLenovo Blue Ocean Research Fund and Beijing Natural Science Foundation under Grant(No.L221003).
文摘Vehicular edge computing(VEC)is emerging as a promising solution paradigm to meet the requirements of compute-intensive applications in internet of vehicle(IoV).Non-orthogonal multiple access(NOMA)has advantages in improving spectrum efficiency and dealing with bandwidth scarcity and cost.It is an encouraging progress combining VEC and NOMA.In this paper,we jointly optimize task offloading decision and resource allocation to maximize the service utility of the NOMA-VEC system.To solve the optimization problem,we propose a multiagent deep graph reinforcement learning algorithm.The algorithm extracts the topological features and relationship information between agents from the system state as observations,outputs task offloading decision and resource allocation simultaneously with local policy network,which is updated by a local learner.Simulation results demonstrate that the proposed method achieves a 1.52%∼5.80%improvement compared with the benchmark algorithms in system service utility.
基金supported by Institute of Information&Communications Technology Planning and Evaluation(IITP)grant funded by the Korean government(MSIT)(No.RS-2022-00167197,Development of Intelligent 5G/6G Infrastructure Technology for the Smart City)in part by the National Research Foundation of Korea(NRF),Ministry of Education,through the Basic Science Research Program under Grant NRF-2020R1I1A3066543+1 种基金in part by BK21 FOUR(Fostering Outstanding Universities for Research)under Grant 5199990914048in part by the Soonchunhyang University Research Fund.
文摘Recently,Network Functions Virtualization(NFV)has become a critical resource for optimizing capability utilization in the 5G/B5G era.NFV decomposes the network resource paradigm,demonstrating the efficient utilization of Network Functions(NFs)to enable configurable service priorities and resource demands.Telecommunications Service Providers(TSPs)face challenges in network utilization,as the vast amounts of data generated by the Internet of Things(IoT)overwhelm existing infrastructures.IoT applications,which generate massive volumes of diverse data and require real-time communication,contribute to bottlenecks and congestion.In this context,Multiaccess Edge Computing(MEC)is employed to support resource and priority-aware IoT applications by implementing Virtual Network Function(VNF)sequences within Service Function Chaining(SFC).This paper proposes the use of Deep Reinforcement Learning(DRL)combined with Graph Neural Networks(GNN)to enhance network processing,performance,and resource pooling capabilities.GNN facilitates feature extraction through Message-Passing Neural Network(MPNN)mechanisms.Together with DRL,Deep Q-Networks(DQN)are utilized to dynamically allocate resources based on IoT network priorities and demands.Our focus is on minimizing delay times for VNF instance execution,ensuring effective resource placement,and allocation in SFC deployments,offering flexibility to adapt to real-time changes in priority and workload.Simulation results demonstrate that our proposed scheme outperforms reference models in terms of reward,delay,delivery,service drop ratios,and average completion ratios,proving its potential for IoT applications.
基金supported by the National Science and Technology Major Project(2021ZD0112702)the National Natural Science Foundation(NNSF)of China(62373100,62233003)the Natural Science Foundation of Jiangsu Province of China(BK20202006)。
文摘This article studies the effective traffic signal control problem of multiple intersections in a city-level traffic system.A novel regional multi-agent cooperative reinforcement learning algorithm called RegionSTLight is proposed to improve the traffic efficiency.Firstly a regional multi-agent Q-learning framework is proposed,which can equivalently decompose the global Q value of the traffic system into the local values of several regions Based on the framework and the idea of human-machine cooperation,a dynamic zoning method is designed to divide the traffic network into several strong-coupled regions according to realtime traffic flow densities.In order to achieve better cooperation inside each region,a lightweight spatio-temporal fusion feature extraction network is designed.The experiments in synthetic real-world and city-level scenarios show that the proposed RegionS TLight converges more quickly,is more stable,and obtains better asymptotic performance compared to state-of-theart models.
基金supported in part by the Natural Science Foundation of China under Grant 61972169 and U1536203in part by the National key research and developm program of China(2016QY01W0200)in part by the Major Scientic and Technological Project of Hubei Province(2018AAA068 and 2019AAA051).
文摘The majority of existing graph-network-based few-shot models focus on a node-similarity update mode.The lack of adequate information intensies the risk of overtraining.In this paper,we propose a novel Multihead Attention Graph Network to excavate discriminative relation and fulll effective information propagation.For edge update,the node-level attention is used to evaluate the similarities between the two nodes and the distributionlevel attention extracts more in-deep global relation.The cooperation between those two parts provides a discriminative and comprehensive expression for edge feature.For node update,we embrace the label-level attention to soften the noise of irrelevant nodes and optimize the update direction.Our proposed model is veried through extensive experiments on two few-shot benchmark MiniImageNet and CIFAR-FS dataset.The results suggest that our method has a strong capability of noise immunity and quick convergence.The classication accuracy outperforms most state-of-the-art approaches.
基金The financial support fromthe Major Science and Technology Programs inHenan Province(Grant No.241100210100)National Natural Science Foundation of China(Grant No.62102372)+3 种基金Henan Provincial Department of Science and Technology Research Project(Grant No.242102211068)Henan Provincial Department of Science and Technology Research Project(Grant No.232102210078)the Stabilization Support Program of The Shenzhen Science and Technology Innovation Commission(Grant No.20231130110921001)the Key Scientific Research Project of Higher Education Institutions of Henan Province(Grant No.24A520042)is acknowledged.
文摘Aiming at the rapid growth of network services,which leads to the problems of long service request processing time and high deployment cost in the deployment of network function virtualization service function chain(SFC)under 5G networks,this paper proposes a multi-agent deep deterministic policy gradient optimization algorithm for SFC deployment(MADDPG-SD).Initially,an optimization model is devised to enhance the request acceptance rate,minimizing the latency and deploying the cost SFC is constructed for the network resource-constrained case.Subsequently,we model the dynamic problem as a Markov decision process(MDP),facilitating adaptation to the evolving states of network resources.Finally,by allocating SFCs to different agents and adopting a collaborative deployment strategy,each agent aims to maximize the request acceptance rate or minimize latency and costs.These agents learn strategies from historical data of virtual network functions in SFCs to guide server node selection,and achieve approximately optimal SFC deployment strategies through a cooperative framework of centralized training and distributed execution.Experimental simulation results indicate that the proposed method,while simultaneously meeting performance requirements and resource capacity constraints,has effectively increased the acceptance rate of requests compared to the comparative algorithms,reducing the end-to-end latency by 4.942%and the deployment cost by 8.045%.
基金National Key Research and Development Program of China,Grant/Award Number:2021YFB1714700Postdoctoral Research Foundation of China,Grant/Award Number:2024M752364Postdoctoral Fellowship Program of CPSF,Grant/Award Number:GZB20240525。
文摘This paper focuses on the problem of multi-station multi-robot spot welding task assignment,and proposes a deep reinforcement learning(DRL)framework,which is made up of a public graph attention network and independent policy networks.The graph of welding spots distribution is encoded using the graph attention network.Independent policy networks with attention mechanism as a decoder can handle the encoded graph and decide to assign robots to different tasks.The policy network is used to convert the large scale welding spots allocation problem to multiple small scale singlerobot welding path planning problems,and the path planning problem is quickly solved through existing methods.Then,the model is trained through reinforcement learning.In addition,the task balancing method is used to allocate tasks to multiple stations.The proposed algorithm is compared with classical algorithms,and the results show that the algorithm based on DRL can produce higher quality solutions.
基金supported by the Innovative Research Group Project of the National Natural Science Foundation of China(22021004)Sinopec Major Science and Technology Projects(321123-1).
文摘Anomaly fluctuations in operating conditions, catalyst wear, crushing, and the deterioration of feedstock properties in fluid catalytic cracking (FCC) units can disrupt the normal circulating fluidization process of the catalyst. Although several effective models have been proposed in previous research to address anomaly detection in chemical processes, most fail to adequately capture the spatial-temporal dependencies of multi-source, mixed-frequency information. In this study, an innovative multi-source mixed-frequency information fusion framework based on a spatial-temporal graph attention network (MIF-STGAT) is proposed to investigate the causes of FCC regenerator catalyst loss anomalies for guide onsite operational management, enhancing the long-term stability of FCC unit operations. First, a reconstruction-based dual-encoder-decoder framework is developed to facilitate the acquisition of mixed-frequency features and information fusion during the FCC regenerator catalyst loss process. Subsequently, a graph attention network and a multilayer long short-term memory network with a differential structure are integrated into the reconstruction-based dual-encoder-shared-decoder framework to capture the dynamic fluctuations and critical features associated with anomalies. Experimental results from the Chinese FCC industrial process demonstrate that MIF-STGAT achieves excellent accuracy and interpretability for anomaly detection.
基金supported by the Vietnam Ministry of Education and Training under project code B2023-SPH-14。
文摘It is known that long non-coding RNAs(lncRNAs)play vital roles in biological processes and contribute to the progression,development,and treatment of various diseases.Obviously,understanding associations between diseases and lncRNAs significantly enhances our ability to interpret disease mechanisms.Nevertheless,the process of determining lncRNA-disease associations is costly,labor-intensive,and time-consuming.Hence,it is expected to foster computational strategies to uncover lncRNA-disease relationships for further verification to save time and resources.In this study,a collaborative filtering and graph attention network-based LncRNA-Disease Association(CFGANLDA)method was nominated to expose potential lncRNA-disease associations.First,it takes into account the advantages of using biological information from multiple sources.Next,it uses a collaborative filtering technique in order to address the sparse data problem.It also employs a graph attention network to reinforce both linear and non-linear features of the associations to advance prediction performance.The computational results indicate that CFGANLDA gains better prediction performance compared to other state-of-the-art approaches.The CFGANLDA’s area under the receiver operating characteristic curve(AUC)metric is 0.9835,whereas its area under the precision-recall curve(AUPR)metric is 0.9822.Statistical analysis using 10-fold cross-validation experiments proves that these metrics are significant.Furthermore,three case studies on prostate,liver,and stomach cancers attest to the validity of CFGANLDA performance.As a result,CFGANLDA method proves to be a valued tool for lncRNA-disease association prediction.
基金supported by the National Science and Technology Council(NSTC),Taiwan,under Grants Numbers 112-2622-E-029-009 and 112-2221-E-029-019.
文摘In the domain of knowledge graph embedding,conventional approaches typically transform entities and relations into continuous vector spaces.However,parameter efficiency becomes increasingly crucial when dealing with large-scale knowledge graphs that contain vast numbers of entities and relations.In particular,resource-intensive embeddings often lead to increased computational costs,and may limit scalability and adaptability in practical environ-ments,such as in low-resource settings or real-world applications.This paper explores an approach to knowledge graph representation learning that leverages small,reserved entities and relation sets for parameter-efficient embedding.We introduce a hierarchical attention network designed to refine and maximize the representational quality of embeddings by selectively focusing on these reserved sets,thereby reducing model complexity.Empirical assessments validate that our model achieves high performance on the benchmark dataset with fewer parameters and smaller embedding dimensions.The ablation studies further highlight the impact and contribution of each component in the proposed hierarchical attention structure.
基金supported by National Key R&D Program of China(2022YFB3104200)in part by National Natural Science Foundation of China(62202386)+2 种基金in part by Basic Research Programs of Taicang(TC2021JC31)in part by Fundamental Research Funds for the Central Universities(D5000210817)in part by Xi’an Unmanned System Security and Intelligent Communications ISTC Center,and in part by Special Funds for Central Universities Construction of World-Class Universities(Disciplines)and Special Development Guidance(0639022GH0202237 and 0639022SH0201237).
文摘Industrial Internet combines the industrial system with Internet connectivity to build a new manufacturing and service system covering the entire industry chain and value chain.Its highly heterogeneous network structure and diversified application requirements call for the applying of network slicing technology.Guaranteeing robust network slicing is essential for Industrial Internet,but it faces the challenge of complex slice topologies caused by the intricate interaction relationships among Network Functions(NFs)composing the slice.Existing works have not concerned the strengthening problem of industrial network slicing regarding its complex network properties.Towards this end,we aim to study this issue by intelligently selecting a subset of most valuable NFs with the minimum cost to satisfy the strengthening requirements.State-of-the-art AlphaGo series of algorithms and the advanced graph neural network technology are combined to build the solution.Simulation results demonstrate the superior performance of our scheme compared to the benchmark schemes.
基金supported by the Tianjin Science and Technology Program(No.22JCZDJC00820).
文摘The proliferation of distributed energy resources and time-varying network topologies in active distribution networks presents unprecedented challenges for network operators.While reinforcement learning (RL) has shown promise in addressing network-constrained energy scheduling,it faces difficulties in managing the complexities of dynamic topologies and discrete-continuous hybrid action spaces.To address these challenges,a graph-based safe RL approach is proposed to learn dynamic optimal power flow under time-varying network topologies.This proposed approach leverages graph convolution operators to handle network topology changes,while safe RL with parameterized action ensures policy development.Specifically,the graph convolution operator abstracts key characteristics of the network topology,enabling effective power flow management in non-stationary environments.Besides that,a parameterized action constrained Markov decision process is employed to handle the hybrid action space and ensure compliance with physical network constraints,thereby accelerating the deployment of safe policy for hybrid action spaces.Numerical results demonstrate that the proposed approach efficiently navigates the discrete-continuous decision space while accounting for the constraints imposed by the dynamic nature of power flow in time-varying network topologies.
基金supported by China Southern Power Grid Co.,Ltd.Yunnan Electric Power Dispatching Control Center(Cyber-physical-based“Source-network-load-storage”Coordinated Dispatch and Control Technologies and Application System Development,sub-project YNKJXM20222463)。
文摘The high proportion of renewable energy integration and the dynamic changes in grid topology necessitate the enhancement of voltage/var control(VVC)to manage voltage fluctuations more rapidly.Traditional model-based control algorithms are becoming increasingly incompetent for VVC due to their high model dependence and slow online computation speed.To alleviate these issues,this paper introduces a graph attention network(GAT)based deep reinforcement learning for VVC of topologically variable power system.Firstly,combining the physical information of the actual power grid,a physics-informed GAT is proposed and embedded into the proximal policy optimization(PPO)algorithm.The GAT-PPO algorithm can capture topological and spatial correlations among the node features to tackle topology changes.To address the slow training,the Relief F-S algorithm identifies critical state variables,significantly reducing the dimensionality of state space.Then,the training samples retained in the experience buffer are designed to mitigate the sparse reward issue.Finally,the validation on the modified IEEE 39-bus system and an actual power grid demonstrates superior performance of the proposed algorithm compared with state-of-the-art algorithms,including PPO algorithm and twin delayed deep deterministic policy gradient(TD3)algorithm.The proposed algorithm exhibits enhanced convergence during training,faster solution speed,and improved VVC performance,even in scenarios involving grid topology changes and increased renewable energy integration.Meanwhile,in the adopted cases,the network loss is reduced by 6.9%,10.8%,and 7.7%,respectively,demonstrating favorable economic outcomes.
