Partial Differential Equations(PDEs)are model candidates of soft sensing for aero-engine health management units.The existing Physics-Informed Neural Networks(PINNs)have made achievements.However,unmeasurable aero-eng...Partial Differential Equations(PDEs)are model candidates of soft sensing for aero-engine health management units.The existing Physics-Informed Neural Networks(PINNs)have made achievements.However,unmeasurable aero-engine driving sources lead to unknown PDE driving terms,which weaken PINNs feasibility.To this end,Physically Informed Hierarchical Learning followed by Recurrent-Prediction Term(PIHL-RPT)is proposed.First,PIHL is proposed for learning nonhomogeneous PDE solutions,in which two networks NetU and NetG are constructed.NetU is for learning solutions satisfying PDEs;NetG is for learning driving terms to regularize NetU training.Then,we propose a hierarchical learning strategy to optimize and couple NetU and NetG,which are integrated into a data-physics-hybrid loss function.Besides,we prove PIHL-RPT can iteratively generate a series of networks converging to a function,which can approximate a solution to well-posed PDE.Furthermore,RPT is proposed for prediction improvement of PIHL,in which network NetU-RP is constructed to compensate for information loss caused by data sampling and driving sources’immeasurability.Finally,artificial datasets and practical vibration process datasets from our wear experiment platform are used to verify the feasibility and effectiveness of PIHL-RPT based soft sensing.Meanwhile,comparisons with relevant methods,discussions,and PIHL-RPT based health monitoring example are given.展开更多
The satellite orbital pursuit game focuses on studying spacecraft maneuvering strategies in space.Traditional numerical methods often face real-time inadequacies and adaptability limitations when dealing with highly n...The satellite orbital pursuit game focuses on studying spacecraft maneuvering strategies in space.Traditional numerical methods often face real-time inadequacies and adaptability limitations when dealing with highly nonlinear problems.With the advancement of Deep Reinforcement Learning(DRL)technology,continuous-time orbital control capabilities have significantly improved.Despite this,the existing DRL technologies still need adjustments in action delay and discretization structure to better adapt to practical application scenarios.Combining continuous learning and model planning demonstrates the adaptability of these methods in continuous-time decision problems.Additionally,to more effectively handle action delay issues,a new scheduled action execution technique has been developed.This technique optimizes action execution timing through real-time policy adjustments,thus adapting to the dynamic changes in the orbital environment.A Hierarchical Reinforcement Learning(HRL)strategy was also adopted to simplify the decision-making process for long-distance pursuit tasks by setting phased subgoals to gradually approach the target.The effectiveness of the proposed strategy in practical satellite pursuit scenarios has been verified through simulations of two different tasks.展开更多
A group optimal penetration strategy in complex attack and defense confrontation situation is proposed in this paper to solve the coordinated penetration decision-making problem of endo-atmospheric gliding simultaneou...A group optimal penetration strategy in complex attack and defense confrontation situation is proposed in this paper to solve the coordinated penetration decision-making problem of endo-atmospheric gliding simultaneous multi-missile penetration of interceptors.First,the problem of large search space of multi-missile coordinated penetration maneuvers is fully considered,and the flight corridor of multi-missile coordinated penetration is designed to constrain search space of multi-agent coordinated strategy,comprehensively considering path constraints and anticollision constraints of gliding multi-missile flight.Then,a multi-missile hierarchical coordinated decision-making mechanism based on confrontation situation is proposed,and the swarm penetration strategy is optimized with the goal of maximizing swarm penetration effectiveness.The upper layer plans the swarm penetration formation according to confrontation situation,and generates the swarm coordinated penetration trajectory based on Multi-Agent Deep Deterministic Policy Gradient(MADDPG)method.The lower layer interpolates and smooths penetration trajectory,and generates the penetration guidance command based on Soft Actor-Critic and Extended Proportional Guidance(SAC-EPG)method.Simulation results verify that the proposed multi-missile cooperative penetration method based on hierarchical reinforcement learning converges faster than the penetration method based on MADDPG,and can quickly learn multi-missile cooperative penetration skills.In addition,multi-missile coordination can give full play to the group's detection and maneuverability,and occupy favorable penetration time and space through coordinated ballistic maneuvers.Thus the success rate of group penetration can be improved.展开更多
Low visibility conditions,particularly those caused by fog,significantly affect road safety and reduce drivers’ability to see ahead clearly.The conventional approaches used to address this problem primarily rely on i...Low visibility conditions,particularly those caused by fog,significantly affect road safety and reduce drivers’ability to see ahead clearly.The conventional approaches used to address this problem primarily rely on instrument-based and fixed-threshold-based theoretical frameworks,which face challenges in adaptability and demonstrate lower performance under varying environmental conditions.To overcome these challenges,we propose a real-time visibility estimation model that leverages roadside CCTV cameras to monitor and identify visibility levels under different weather conditions.The proposedmethod begins by identifying specific regions of interest(ROI)in the CCTVimages and focuses on extracting specific features such as the number of lines and contours detected within these regions.These features are then provided as an input to the proposed hierarchical clusteringmodel,which classifies them into different visibility levels without the need for predefined rules and threshold values.In the proposed approach,we used two different distance similaritymetrics,namely dynamic time warping(DTW)and Euclidean distance,alongside the proposed hierarchical clustering model and noted its performance in terms of numerous evaluation measures.The proposed model achieved an average accuracy of 97.81%,precision of 91.31%,recall of 91.25%,and F1-score of 91.27% using theDTWdistancemetric.We also conducted experiments for other deep learning(DL)-based models used in the literature and compared their performances with the proposed model.The experimental results demonstrate that the proposedmodel ismore adaptable and consistent compared to themethods used in the literature.The proposedmethod provides drivers real-time and accurate visibility information and enhances road safety during low visibility conditions.展开更多
In reinforcement learning an agent may explore ineffectively when dealing with sparse reward tasks where finding a reward point is difficult.To solve the problem,we propose an algorithm called hierarchical deep reinfo...In reinforcement learning an agent may explore ineffectively when dealing with sparse reward tasks where finding a reward point is difficult.To solve the problem,we propose an algorithm called hierarchical deep reinforcement learning with automatic sub-goal identification via computer vision(HADS)which takes advantage of hierarchical reinforcement learning to alleviate the sparse reward problem and improve efficiency of exploration by utilizing a sub-goal mechanism.HADS uses a computer vision method to identify sub-goals automatically for hierarchical deep reinforcement learning.Due to the fact that not all sub-goal points are reachable,a mechanism is proposed to remove unreachable sub-goal points so as to further improve the performance of the algorithm.HADS involves contour recognition to identify sub-goals from the state image where some salient states in the state image may be recognized as sub-goals,while those that are not will be removed based on prior knowledge.Our experiments verified the effect of the algorithm.展开更多
The guidance strategy is an extremely critical factor in determining the striking effect of the missile operation.A novel guidance law is presented by exploiting the deep reinforcement learning(DRL)with the hierarchic...The guidance strategy is an extremely critical factor in determining the striking effect of the missile operation.A novel guidance law is presented by exploiting the deep reinforcement learning(DRL)with the hierarchical deep deterministic policy gradient(DDPG)algorithm.The reward functions are constructed to minimize the line-of-sight(LOS)angle rate and avoid the threat caused by the opposed obstacles.To attenuate the chattering of the acceleration,a hierarchical reinforcement learning structure and an improved reward function with action penalty are put forward.The simulation results validate that the missile under the proposed method can hit the target successfully and keep away from the threatened areas effectively.展开更多
Federated learning(FL)is a distributed machine learning(ML)framework where several clients cooperatively train an ML model by exchanging the model parameters without directly sharing their local data.In FL,the limited...Federated learning(FL)is a distributed machine learning(ML)framework where several clients cooperatively train an ML model by exchanging the model parameters without directly sharing their local data.In FL,the limited number of participants for model aggregation and communication latency are two major bottlenecks.Hierarchical federated learning(HFL),with a cloud-edge-client hierarchy,can leverage the large coverage of cloud servers and the low transmission latency of edge servers.There are growing research interests in implementing FL in vehicular networks due to the requirements of timely ML training for intelligent vehicles.However,the limited number of participants in vehicular networks and vehicle mobility degrade the performance of FL training.In this context,HFL,which stands out for lower latency,wider coverage and more participants,is promising in vehicular networks.In this paper,we begin with the background and motivation of HFL and the feasibility of implementing HFL in vehicular networks.Then,the architecture of HFL is illustrated.Next,we clarify new issues in HFL and review several existing solutions.Furthermore,we introduce some typical use cases in vehicular networks as well as our initial efforts on implementing HFL in vehicular networks.Finally,we conclude with future research directions.展开更多
Based on option-critic algorithm,a new adversarial algorithm named deterministic policy network with option architecture is proposed to improve agent's performance against opponent with fixed offensive algorithm.A...Based on option-critic algorithm,a new adversarial algorithm named deterministic policy network with option architecture is proposed to improve agent's performance against opponent with fixed offensive algorithm.An option network is introduced in upper level design,which can generate activated signal from defensive and of-fensive strategies according to temporary situation.Then the lower level executive layer can figure out interactive action with guidance of activated signal,and the value of both activated signal and interactive action is evaluated by critic structure together.This method could release requirement of semi Markov decision process effectively and eventually simplified network structure by eliminating termination possibility layer.According to the result of experiment,it is proved that new algorithm switches strategy style between offensive and defensive ones neatly and acquires more reward from environment than classical deep deterministic policy gradient algorithm does.展开更多
In the context of edge computing environments in general and the metaverse in particular,federated learning(FL)has emerged as a distributed machine learning paradigm that allows multiple users to collaborate on traini...In the context of edge computing environments in general and the metaverse in particular,federated learning(FL)has emerged as a distributed machine learning paradigm that allows multiple users to collaborate on training a shared machine learning model locally,eliminating the need for uploading raw data to a central server.It is perhaps the only training paradigm that preserves the privacy of user data,which is essential for computing environments as personal as the metaverse.However,the original FL architecture proposed is not scalable to a large number of user devices in the metaverse community.To mitigate this problem,hierarchical federated learning(HFL)has been introduced as a general distributed learning paradigm,inspiring a number of research works.In this paper,we present several types of HFL architectures,with a special focus on the three-layer client-edge-cloud HFL architecture,which is most pertinent to the metaverse due to its delay-sensitive nature.We also examine works that take advantage of the natural layered organization of three-layer client-edge-cloud HFL to tackle some of the most challenging problems in FL within the metaverse.Finally,we outline some future research directions of HFL in the metaverse.展开更多
Computer-aided design(CAD)software continues to be a crucial tool in digital twin application and manufacturing,facilitating the design of various products.We present a novel CAD generation method,an agent that constr...Computer-aided design(CAD)software continues to be a crucial tool in digital twin application and manufacturing,facilitating the design of various products.We present a novel CAD generation method,an agent that constructs the CAD sequences containing the sketch-and-extrude modelling operations efficiently and with high quality.Starting from the sketch and extrusion operation sequences,we utilise the transformer encoder to encode them into different disentangled codebooks to represent their distribution properties while considering their correlations.Then,a combination of auto-regressive and non-autoregressive samplers is trained to sample the code for CAD sequence con-struction.Extensive experiments demonstrate that our model generates diverse and high-quality CAD models.We also show some cases of real digital twin applications and indicate that our generated model can be used as the data source for the digital twin platform,exhibiting designers'potential.展开更多
Flapping Wing Aerial Vehicles(FWAVs)hold immense potential for applications such as search-and-rescue missions in complex terrains,environmental monitoring in hazardous areas,and exploration in confined spaces.However...Flapping Wing Aerial Vehicles(FWAVs)hold immense potential for applications such as search-and-rescue missions in complex terrains,environmental monitoring in hazardous areas,and exploration in confined spaces.However,their adoption is hindered by the challenges of autonomous navigation in unknown environments,exacerbated by their limited onboard computational resources and demanding flight dynamics.This work addresses these challenges by presenting a lightweight,vision-based autonomous navigation system weighing 26.0 g,enabling FWAVs to achieve obstacle-avoidance flight at a speed of 9.0 m/s.Central to this system is a novel end-toend Bi-level Cooperative Policy(BCP)that significantly improves flight efficiency and safety.BCP employs lightweight neural networks for real-time performance and leverages Hierarchical Reinforcement Learning(HRL)for robust and efficient training.Quantitative evaluations show that BCP achieves up to 6.5%shorter path lengths,11.2%faster task completion time,and improved explainability compared to state-of-the-art reinforcement learning algorithms.Additionally,BCP demonstrates 35.7%more efficient and stable training,reducing computational overhead while maintaining high performance.The system design incorporates optimized lightweight components,including a 4.0 g customized stereo camera,a 6.0 g 3D-printed camera mount,and a 16.0 g onboard computer,all tailored to FWAV applications.Real-flight experiments validate the sim-toreal transferability of the proposed navigation system,demonstrating its readiness for real-world deployment in challenging scenarios.This research advances the practicality of FWAVs,paving the way for their broader adoption in critical missions where compact,agile aerial robots are indispensable.展开更多
Goal-conditioned hierarchical reinforcement learning(GCHRL)decomposes the desired goal into subgoals and conducts exploration and exploitation in the subgoal space.Its effectiveness heavily relies on subgoal represent...Goal-conditioned hierarchical reinforcement learning(GCHRL)decomposes the desired goal into subgoals and conducts exploration and exploitation in the subgoal space.Its effectiveness heavily relies on subgoal representation and selection.However,existing works do not consider distinct information across hierarchical time scales when learning subgoal representations and lack a subgoal selection strategy that balances exploration and exploitation.In this paper,we propose a novel method for efficient exploration-exploitation balance in HIerarchical reinforcement learning by dynamically constructing Latent Landmark graphs(HILL).HILL transforms the reward maximization problem of GCHRL into the shortest path planning on graphs.To effectively consider the hierarchical time-scale information,HILL adopts a contrastive representation learning objective to learn informative latent representations.Based on these representations,HILL dynamically constructs latent landmark graphs and selects subgoals using two measures to balance exploration and exploitation.We implement two variants:HILL-hf generates graphs periodically,while HILL-lf generates graphs adaptively.Empirical results on continuous control tasks with sparse rewards demonstrate that both variants outperform state-of-the-art baselines in sample efficiency and asymptotic performance,with HILL-lf further reducing training time by 40%compared to HILL-hf.展开更多
Hierarchical Federated Learning(HFL)extends traditional Federated Learning(FL)by introducing multi-level aggregation in which model updates pass through clients,edge servers,and a global server.While this hierarchical...Hierarchical Federated Learning(HFL)extends traditional Federated Learning(FL)by introducing multi-level aggregation in which model updates pass through clients,edge servers,and a global server.While this hierarchical structure enhances scalability,it also increases vulnerability to adversarial attacks—such as data poisoning and model poisoning—that disrupt learning by introducing discrepancies at the edge server level.These discrepancies propagate through aggregation,affecting model consistency and overall integrity.Existing studies on adversarial behaviour in FL primarily rely on single-metric approaches—such as cosine similarity or Euclidean distance—to assess model discrepancies and filter out anomalous updates.However,these methods fail to capture the diverse ways adversarial attacks influence model updates,particularly in highly heterogeneous data environments and hierarchical structures.Attackers can exploit the limitations of single-metric defences by crafting updates that seem benign under one metric while remaining anomalous under another.Moreover,prior studies have not systematically analysed how model discrepancies evolve over time,vary across regions,or affect clustering structures in HFL architectures.To address these limitations,we propose the Model Discrepancy Score(MDS),a multi-metric framework that integrates Dissimilarity,Distance,Uncorrelation,and Divergence to provide a comprehensive analysis of how adversarial activity affects model discrepancies.Through temporal,spatial,and clustering analyses,we examine how attacks affect model discrepancies at the edge server level in 3LHFL and 4LHFL architectures and evaluate MDS’s ability to distinguish between benign and malicious servers.Our results show that while 4LHFL effectively mitigates discrepancies in regional attack scenarios,it struggles with distributed attacks due to additional aggregation layers that obscure distinguishable discrepancy patterns over time,across regions,and within clustering structures.Factors influencing detection include data heterogeneity,attack sophistication,and hierarchical aggregation depth.These findings highlight the limitations of single-metric approaches and emphasize the need for multi-metric strategies such as MDS to enhance HFL security.展开更多
To improve the driving efficiency of unmanned vehicles in a complex urban traffic flow environment and the safety and passenger comfort of vehicles when changing lanes,we propose a hierarchical reinforcement learning(...To improve the driving efficiency of unmanned vehicles in a complex urban traffic flow environment and the safety and passenger comfort of vehicles when changing lanes,we propose a hierarchical reinforcement learning(HRL)-based vehicle trajectory planning and tracking method.First,we present a hierarchical control framework for vehicle trajectory tracking that is based on deep reinforcement learning(DRL)and model predictive control(MPC).We design an upper-level decision model based on the trust region policy optimization algorithm integrated with long short-term memory to obtain more accurate strategies.Second,to improve stability and passenger comfort,we constructed a lower controller that combines the Bezier curve fitting method and an MPC controller.Finally,the proposed method was simulated via the car learning to act(CARLA)simulator,which is based on an unreal engine.Random urban traffic-flow test scenarios were used to simulate a real urban road-traffic environment.The simulation results illustrate that the proposed method can complete the vehicle trajectory planning and tracking task well.Compared with the existing RL methods,our proposed method has the lowest collision rate of 1.5%and achieves an average speed improvement of 7.04%.Moreover,our proposed method has better comfort performance and lower fuel consumption during the driving process.展开更多
In recent research on the Digital Twin-based Vehicular Ad hoc Network(DT-VANET),Federated Learning(FL)has shown its ability to provide data privacy.However,Federated learning struggles to adequately train a global mod...In recent research on the Digital Twin-based Vehicular Ad hoc Network(DT-VANET),Federated Learning(FL)has shown its ability to provide data privacy.However,Federated learning struggles to adequately train a global model when confronted with data heterogeneity and data sparsity among vehicles,which ensure suboptimal accuracy in making precise predictions for different vehicle types.To address these challenges,this paper combines Federated Transfer Learning(FTL)to conduct vehicle clustering related to types of vehicles and proposes a novel Hierarchical Federated Transfer Learning(HFTL).We construct a framework for DT-VANET,along with two algorithms designed for cloud server model updates and intra-cluster federated transfer learning,to improve the accuracy of the global model.In addition,we developed a data quality score-based mechanism to prevent the global model from being affected by malicious vehicles.Lastly,detailed experiments on real-world datasets are conducted,considering different performance metrics that verify the effectiveness and efficiency of our algorithm.展开更多
As intelligent vehicles usually have complex overtaking process,a safe and efficient automated overtaking system(AOS)is vital to avoid accidents caused by wrong operation of drivers.Existing AOSs rarely consider longi...As intelligent vehicles usually have complex overtaking process,a safe and efficient automated overtaking system(AOS)is vital to avoid accidents caused by wrong operation of drivers.Existing AOSs rarely consider longitudinal reactions of the overtaken vehicle(OV)during overtaking.This paper proposed a novel AOS based on hierarchical reinforcement learning,where the longitudinal reaction is given by a data-driven social preference estimation.This AOS incorporates two modules that can function in different overtaking phases.The first module based on semi-Markov decision process and motion primitives is built for motion planning and control.The second module based on Markov decision process is designed to enable vehicles to make proper decisions according to the social preference of OV.Based on realistic overtaking data,the proposed AOS and its modules are verified experimentally.The results of the tests show that the proposed AOS can realize safe and effective overtaking in scenes built by realistic data,and has the ability to flexibly adjust lateral driving behavior and lane changing position when the OVs have different social preferences.展开更多
Option is a promising method to discover the hierarchical structure in reinforcement learning （RL） for learning acceleration. The key to option discovery is about how an agent can find useful subgoals autonomically ...Option is a promising method to discover the hierarchical structure in reinforcement learning （RL） for learning acceleration. The key to option discovery is about how an agent can find useful subgoals autonomically among the passing trails. By analyzing the agent＇s actions in the trails, useful heuristics can be found. Not only does the agent pass subgoals more frequently, but also its effective actions are restricted in subgoals. As a consequence, the subgoals can be deemed as the most matching action-restricted states in the paths. In the grid-world environment, the concept of the unique-direction value reflecting the action-restricted property was introduced to find the most matching action-restricted states. The unique-direction-value （UDV） approach is chosen to form options offline and online autonomically. Experiments show that the approach can find subgoals correctly. Thus the Q-learning with options found on both offline and online process can accelerate learning significantly.展开更多
In some image classification tasks, similarities among different categories are different and the samples are usually misclassified as highly similar categories. To distinguish highly similar categories, more specific...In some image classification tasks, similarities among different categories are different and the samples are usually misclassified as highly similar categories. To distinguish highly similar categories, more specific features are required so that the classifier can improve the classification performance. In this paper, we propose a novel two-level hierarchical feature learning framework based on the deep convolutional neural network(CNN), which is simple and effective. First, the deep feature extractors of different levels are trained using the transfer learning method that fine-tunes the pre-trained deep CNN model toward the new target dataset. Second, the general feature extracted from all the categories and the specific feature extracted from highly similar categories are fused into a feature vector. Then the final feature representation is fed into a linear classifier. Finally, experiments using the Caltech-256, Oxford Flower-102, and Tasmania Coral Point Count(CPC) datasets demonstrate that the expression ability of the deep features resulting from two-level hierarchical feature learning is powerful. Our proposed method effectively increases the classification accuracy in comparison with flat multiple classification methods.展开更多
This study proposed a weighted sampling hierarchical classification learning method based on an efficient backbone network model to address the problems of high costs,low accuracy,and time-consuming traditional tea di...This study proposed a weighted sampling hierarchical classification learning method based on an efficient backbone network model to address the problems of high costs,low accuracy,and time-consuming traditional tea disease recognition methods.This method enhances the feature extraction ability by conducting hierarchical classification learning based on the EfficientNet model,effectively alleviating the impact of high similarity between tea diseases on the model’s classification performance.To better solve the problem of few and unevenly distributed tea disease samples,this study introduced a weighted sampling scheme to optimize data processing,which not only alleviates the overfitting effect caused by too few sample data but also balances the probability of extracting imbalanced classification data.The experimental results show that the proposed method was significant in identifying both healthy tea leaves and four common leaf diseases of tea(tea algal spot disease,tea white spot disease,tea anthracnose disease,and tea leaf blight disease).After applying the“weighted sampling hierarchical classification learning method”to train 7 different efficient backbone networks,most of their accuracies have improved.The EfficientNet-B1 model proposed in this study achieved an accuracy rate of 99.21%after adopting this learning method,which is higher than EfficientNet-b2(98.82%)and MobileNet-V3(98.43%).In addition,to better apply the results of identifying tea diseases,this study developed a mini-program that operates on WeChat.Users can quickly obtain accurate identification results and corresponding disease descriptions and prevention methods through simple operations.This intelligent tool for identifying tea diseases can serve as an auxiliary tool for farmers,consumers,and related scientific researchers and has certain practical value.展开更多
Shuttle tankers scheduling is an important task in offshore oil and gas transportation process,which involves operating time window fulfillment,optimal transportation planning,and proper inventory management.However,c...Shuttle tankers scheduling is an important task in offshore oil and gas transportation process,which involves operating time window fulfillment,optimal transportation planning,and proper inventory management.However,conventional approaches like Mixed lnteger Linear Programming(MlLP)or meta heuristic algorithms often fail in long running time.In this paper,a Graph Pointer Network(GPN)based Hierarchical Curriculum Reinforcement Learning(HCRl)method is proposed to solve Shuttle Tankers Scheduling Problem(STSP)The model is trained to divide STSP into voyage and operation stages and generate routing and inventory management decisions sequentially.An asynchronous training strategy is developed to address the coupling between stages.Comparison experiments demonstrate that the proposed HCRL method achieves 12%shortel tour lengths on average compared to heuristic algorithms.Additional experiments validate its generalizability to unseen instances and scalability to larger instances.展开更多
基金supported in part by the National Science and Technology Major Project of China(No.2019-I-0019-0018)the National Natural Science Foundation of China(Nos.61890920,61890921,12302065 and 12172073).
文摘Partial Differential Equations(PDEs)are model candidates of soft sensing for aero-engine health management units.The existing Physics-Informed Neural Networks(PINNs)have made achievements.However,unmeasurable aero-engine driving sources lead to unknown PDE driving terms,which weaken PINNs feasibility.To this end,Physically Informed Hierarchical Learning followed by Recurrent-Prediction Term(PIHL-RPT)is proposed.First,PIHL is proposed for learning nonhomogeneous PDE solutions,in which two networks NetU and NetG are constructed.NetU is for learning solutions satisfying PDEs;NetG is for learning driving terms to regularize NetU training.Then,we propose a hierarchical learning strategy to optimize and couple NetU and NetG,which are integrated into a data-physics-hybrid loss function.Besides,we prove PIHL-RPT can iteratively generate a series of networks converging to a function,which can approximate a solution to well-posed PDE.Furthermore,RPT is proposed for prediction improvement of PIHL,in which network NetU-RP is constructed to compensate for information loss caused by data sampling and driving sources’immeasurability.Finally,artificial datasets and practical vibration process datasets from our wear experiment platform are used to verify the feasibility and effectiveness of PIHL-RPT based soft sensing.Meanwhile,comparisons with relevant methods,discussions,and PIHL-RPT based health monitoring example are given.
基金supported by the National Natural Science Foundation of China(No.12202281)the Shanghai Natural Science Foundation,China(No.23ZR1461800)the Research Initiation Fund of Northwestern Polytechnical University,China(No.G2024KY05103)。
文摘The satellite orbital pursuit game focuses on studying spacecraft maneuvering strategies in space.Traditional numerical methods often face real-time inadequacies and adaptability limitations when dealing with highly nonlinear problems.With the advancement of Deep Reinforcement Learning(DRL)technology,continuous-time orbital control capabilities have significantly improved.Despite this,the existing DRL technologies still need adjustments in action delay and discretization structure to better adapt to practical application scenarios.Combining continuous learning and model planning demonstrates the adaptability of these methods in continuous-time decision problems.Additionally,to more effectively handle action delay issues,a new scheduled action execution technique has been developed.This technique optimizes action execution timing through real-time policy adjustments,thus adapting to the dynamic changes in the orbital environment.A Hierarchical Reinforcement Learning(HRL)strategy was also adopted to simplify the decision-making process for long-distance pursuit tasks by setting phased subgoals to gradually approach the target.The effectiveness of the proposed strategy in practical satellite pursuit scenarios has been verified through simulations of two different tasks.
文摘A group optimal penetration strategy in complex attack and defense confrontation situation is proposed in this paper to solve the coordinated penetration decision-making problem of endo-atmospheric gliding simultaneous multi-missile penetration of interceptors.First,the problem of large search space of multi-missile coordinated penetration maneuvers is fully considered,and the flight corridor of multi-missile coordinated penetration is designed to constrain search space of multi-agent coordinated strategy,comprehensively considering path constraints and anticollision constraints of gliding multi-missile flight.Then,a multi-missile hierarchical coordinated decision-making mechanism based on confrontation situation is proposed,and the swarm penetration strategy is optimized with the goal of maximizing swarm penetration effectiveness.The upper layer plans the swarm penetration formation according to confrontation situation,and generates the swarm coordinated penetration trajectory based on Multi-Agent Deep Deterministic Policy Gradient(MADDPG)method.The lower layer interpolates and smooths penetration trajectory,and generates the penetration guidance command based on Soft Actor-Critic and Extended Proportional Guidance(SAC-EPG)method.Simulation results verify that the proposed multi-missile cooperative penetration method based on hierarchical reinforcement learning converges faster than the penetration method based on MADDPG,and can quickly learn multi-missile cooperative penetration skills.In addition,multi-missile coordination can give full play to the group's detection and maneuverability,and occupy favorable penetration time and space through coordinated ballistic maneuvers.Thus the success rate of group penetration can be improved.
文摘Low visibility conditions,particularly those caused by fog,significantly affect road safety and reduce drivers’ability to see ahead clearly.The conventional approaches used to address this problem primarily rely on instrument-based and fixed-threshold-based theoretical frameworks,which face challenges in adaptability and demonstrate lower performance under varying environmental conditions.To overcome these challenges,we propose a real-time visibility estimation model that leverages roadside CCTV cameras to monitor and identify visibility levels under different weather conditions.The proposedmethod begins by identifying specific regions of interest(ROI)in the CCTVimages and focuses on extracting specific features such as the number of lines and contours detected within these regions.These features are then provided as an input to the proposed hierarchical clusteringmodel,which classifies them into different visibility levels without the need for predefined rules and threshold values.In the proposed approach,we used two different distance similaritymetrics,namely dynamic time warping(DTW)and Euclidean distance,alongside the proposed hierarchical clustering model and noted its performance in terms of numerous evaluation measures.The proposed model achieved an average accuracy of 97.81%,precision of 91.31%,recall of 91.25%,and F1-score of 91.27% using theDTWdistancemetric.We also conducted experiments for other deep learning(DL)-based models used in the literature and compared their performances with the proposed model.The experimental results demonstrate that the proposedmodel ismore adaptable and consistent compared to themethods used in the literature.The proposedmethod provides drivers real-time and accurate visibility information and enhances road safety during low visibility conditions.
基金supported by the National Natural Science Foundation of China(61303108)Suzhou Key Industries Technological Innovation-Prospective Applied Research Project(SYG201804)+2 种基金A Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions(PAPD)the Fundamental Research Funds for the Gentral UniversitiesJLU(93K172020K25)。
文摘In reinforcement learning an agent may explore ineffectively when dealing with sparse reward tasks where finding a reward point is difficult.To solve the problem,we propose an algorithm called hierarchical deep reinforcement learning with automatic sub-goal identification via computer vision(HADS)which takes advantage of hierarchical reinforcement learning to alleviate the sparse reward problem and improve efficiency of exploration by utilizing a sub-goal mechanism.HADS uses a computer vision method to identify sub-goals automatically for hierarchical deep reinforcement learning.Due to the fact that not all sub-goal points are reachable,a mechanism is proposed to remove unreachable sub-goal points so as to further improve the performance of the algorithm.HADS involves contour recognition to identify sub-goals from the state image where some salient states in the state image may be recognized as sub-goals,while those that are not will be removed based on prior knowledge.Our experiments verified the effect of the algorithm.
基金supported by the National Natural Science Foundation of China(62003021,91212304).
文摘The guidance strategy is an extremely critical factor in determining the striking effect of the missile operation.A novel guidance law is presented by exploiting the deep reinforcement learning(DRL)with the hierarchical deep deterministic policy gradient(DDPG)algorithm.The reward functions are constructed to minimize the line-of-sight(LOS)angle rate and avoid the threat caused by the opposed obstacles.To attenuate the chattering of the acceleration,a hierarchical reinforcement learning structure and an improved reward function with action penalty are put forward.The simulation results validate that the missile under the proposed method can hit the target successfully and keep away from the threatened areas effectively.
基金sponsored in part by the National Key R&D Program of China under Grant No. 2020YFB1806605the National Natural Science Foundation of China under Grant Nos. 62022049, 62111530197, and 61871254+1 种基金OPPOsupported by the Fundamental Research Funds for the Central Universities under Grant No. 2022JBXT001
文摘Federated learning(FL)is a distributed machine learning(ML)framework where several clients cooperatively train an ML model by exchanging the model parameters without directly sharing their local data.In FL,the limited number of participants for model aggregation and communication latency are two major bottlenecks.Hierarchical federated learning(HFL),with a cloud-edge-client hierarchy,can leverage the large coverage of cloud servers and the low transmission latency of edge servers.There are growing research interests in implementing FL in vehicular networks due to the requirements of timely ML training for intelligent vehicles.However,the limited number of participants in vehicular networks and vehicle mobility degrade the performance of FL training.In this context,HFL,which stands out for lower latency,wider coverage and more participants,is promising in vehicular networks.In this paper,we begin with the background and motivation of HFL and the feasibility of implementing HFL in vehicular networks.Then,the architecture of HFL is illustrated.Next,we clarify new issues in HFL and review several existing solutions.Furthermore,we introduce some typical use cases in vehicular networks as well as our initial efforts on implementing HFL in vehicular networks.Finally,we conclude with future research directions.
基金the National Natural Science Foundation of China (No.61673265)the National Key Research and Development Program (No.2020YFC1512203)the Shanghai Commercial Aircraft System Engineering Joint Research Fund (No.CASEF-2022-Z05)。
文摘Based on option-critic algorithm,a new adversarial algorithm named deterministic policy network with option architecture is proposed to improve agent's performance against opponent with fixed offensive algorithm.An option network is introduced in upper level design,which can generate activated signal from defensive and of-fensive strategies according to temporary situation.Then the lower level executive layer can figure out interactive action with guidance of activated signal,and the value of both activated signal and interactive action is evaluated by critic structure together.This method could release requirement of semi Markov decision process effectively and eventually simplified network structure by eliminating termination possibility layer.According to the result of experiment,it is proved that new algorithm switches strategy style between offensive and defensive ones neatly and acquires more reward from environment than classical deep deterministic policy gradient algorithm does.
文摘In the context of edge computing environments in general and the metaverse in particular,federated learning(FL)has emerged as a distributed machine learning paradigm that allows multiple users to collaborate on training a shared machine learning model locally,eliminating the need for uploading raw data to a central server.It is perhaps the only training paradigm that preserves the privacy of user data,which is essential for computing environments as personal as the metaverse.However,the original FL architecture proposed is not scalable to a large number of user devices in the metaverse community.To mitigate this problem,hierarchical federated learning(HFL)has been introduced as a general distributed learning paradigm,inspiring a number of research works.In this paper,we present several types of HFL architectures,with a special focus on the three-layer client-edge-cloud HFL architecture,which is most pertinent to the metaverse due to its delay-sensitive nature.We also examine works that take advantage of the natural layered organization of three-layer client-edge-cloud HFL to tackle some of the most challenging problems in FL within the metaverse.Finally,we outline some future research directions of HFL in the metaverse.
基金National Key Research and Development Program of China,Grant/Award Number:2022YFF0904303Beijing Science and Technology Planning Project,Grant/Award Number:Z221100006322003National Natural Science Foundation of China,Grant/Award Number:61932003。
文摘Computer-aided design(CAD)software continues to be a crucial tool in digital twin application and manufacturing,facilitating the design of various products.We present a novel CAD generation method,an agent that constructs the CAD sequences containing the sketch-and-extrude modelling operations efficiently and with high quality.Starting from the sketch and extrusion operation sequences,we utilise the transformer encoder to encode them into different disentangled codebooks to represent their distribution properties while considering their correlations.Then,a combination of auto-regressive and non-autoregressive samplers is trained to sample the code for CAD sequence con-struction.Extensive experiments demonstrate that our model generates diverse and high-quality CAD models.We also show some cases of real digital twin applications and indicate that our generated model can be used as the data source for the digital twin platform,exhibiting designers'potential.
基金supported by the Fundamental Research Funds for the Central Universities,China。
文摘Flapping Wing Aerial Vehicles(FWAVs)hold immense potential for applications such as search-and-rescue missions in complex terrains,environmental monitoring in hazardous areas,and exploration in confined spaces.However,their adoption is hindered by the challenges of autonomous navigation in unknown environments,exacerbated by their limited onboard computational resources and demanding flight dynamics.This work addresses these challenges by presenting a lightweight,vision-based autonomous navigation system weighing 26.0 g,enabling FWAVs to achieve obstacle-avoidance flight at a speed of 9.0 m/s.Central to this system is a novel end-toend Bi-level Cooperative Policy(BCP)that significantly improves flight efficiency and safety.BCP employs lightweight neural networks for real-time performance and leverages Hierarchical Reinforcement Learning(HRL)for robust and efficient training.Quantitative evaluations show that BCP achieves up to 6.5%shorter path lengths,11.2%faster task completion time,and improved explainability compared to state-of-the-art reinforcement learning algorithms.Additionally,BCP demonstrates 35.7%more efficient and stable training,reducing computational overhead while maintaining high performance.The system design incorporates optimized lightweight components,including a 4.0 g customized stereo camera,a 6.0 g 3D-printed camera mount,and a 16.0 g onboard computer,all tailored to FWAV applications.Real-flight experiments validate the sim-toreal transferability of the proposed navigation system,demonstrating its readiness for real-world deployment in challenging scenarios.This research advances the practicality of FWAVs,paving the way for their broader adoption in critical missions where compact,agile aerial robots are indispensable.
基金supported by National Key R&D Program of China(No.2022ZD0116405)the Strategic Priority Research Program of the Chinese Academy of Sciences,China(No.XDA27030300).
文摘Goal-conditioned hierarchical reinforcement learning(GCHRL)decomposes the desired goal into subgoals and conducts exploration and exploitation in the subgoal space.Its effectiveness heavily relies on subgoal representation and selection.However,existing works do not consider distinct information across hierarchical time scales when learning subgoal representations and lack a subgoal selection strategy that balances exploration and exploitation.In this paper,we propose a novel method for efficient exploration-exploitation balance in HIerarchical reinforcement learning by dynamically constructing Latent Landmark graphs(HILL).HILL transforms the reward maximization problem of GCHRL into the shortest path planning on graphs.To effectively consider the hierarchical time-scale information,HILL adopts a contrastive representation learning objective to learn informative latent representations.Based on these representations,HILL dynamically constructs latent landmark graphs and selects subgoals using two measures to balance exploration and exploitation.We implement two variants:HILL-hf generates graphs periodically,while HILL-lf generates graphs adaptively.Empirical results on continuous control tasks with sparse rewards demonstrate that both variants outperform state-of-the-art baselines in sample efficiency and asymptotic performance,with HILL-lf further reducing training time by 40%compared to HILL-hf.
基金supported by the Technical and Vocational Training Corporation(TVTC)through the Saudi Arabian Culture Bureau(SACB)in the United Kingdom and the EPSRC-funded project National Edge AI Hub for Real Data:Edge Intelligence for Cyber-disturbances and Data Quality(EP/Y028813/1).
文摘Hierarchical Federated Learning(HFL)extends traditional Federated Learning(FL)by introducing multi-level aggregation in which model updates pass through clients,edge servers,and a global server.While this hierarchical structure enhances scalability,it also increases vulnerability to adversarial attacks—such as data poisoning and model poisoning—that disrupt learning by introducing discrepancies at the edge server level.These discrepancies propagate through aggregation,affecting model consistency and overall integrity.Existing studies on adversarial behaviour in FL primarily rely on single-metric approaches—such as cosine similarity or Euclidean distance—to assess model discrepancies and filter out anomalous updates.However,these methods fail to capture the diverse ways adversarial attacks influence model updates,particularly in highly heterogeneous data environments and hierarchical structures.Attackers can exploit the limitations of single-metric defences by crafting updates that seem benign under one metric while remaining anomalous under another.Moreover,prior studies have not systematically analysed how model discrepancies evolve over time,vary across regions,or affect clustering structures in HFL architectures.To address these limitations,we propose the Model Discrepancy Score(MDS),a multi-metric framework that integrates Dissimilarity,Distance,Uncorrelation,and Divergence to provide a comprehensive analysis of how adversarial activity affects model discrepancies.Through temporal,spatial,and clustering analyses,we examine how attacks affect model discrepancies at the edge server level in 3LHFL and 4LHFL architectures and evaluate MDS’s ability to distinguish between benign and malicious servers.Our results show that while 4LHFL effectively mitigates discrepancies in regional attack scenarios,it struggles with distributed attacks due to additional aggregation layers that obscure distinguishable discrepancy patterns over time,across regions,and within clustering structures.Factors influencing detection include data heterogeneity,attack sophistication,and hierarchical aggregation depth.These findings highlight the limitations of single-metric approaches and emphasize the need for multi-metric strategies such as MDS to enhance HFL security.
基金supported in part by the Jiaxing Public Welfare Research Program(Grant No.2023AY11034)the Zhejiang Provincial Natural Science Foundation of China under(Grant No.LTGS23F030002)+1 种基金the National Natural Science Foundation of China(Grant No.61603154)the Open Research Project of the State Key Laboratory of Industrial Control Technology,Zhejiang University,China(Grant No.ICT2022B52).
文摘To improve the driving efficiency of unmanned vehicles in a complex urban traffic flow environment and the safety and passenger comfort of vehicles when changing lanes,we propose a hierarchical reinforcement learning(HRL)-based vehicle trajectory planning and tracking method.First,we present a hierarchical control framework for vehicle trajectory tracking that is based on deep reinforcement learning(DRL)and model predictive control(MPC).We design an upper-level decision model based on the trust region policy optimization algorithm integrated with long short-term memory to obtain more accurate strategies.Second,to improve stability and passenger comfort,we constructed a lower controller that combines the Bezier curve fitting method and an MPC controller.Finally,the proposed method was simulated via the car learning to act(CARLA)simulator,which is based on an unreal engine.Random urban traffic-flow test scenarios were used to simulate a real urban road-traffic environment.The simulation results illustrate that the proposed method can complete the vehicle trajectory planning and tracking task well.Compared with the existing RL methods,our proposed method has the lowest collision rate of 1.5%and achieves an average speed improvement of 7.04%.Moreover,our proposed method has better comfort performance and lower fuel consumption during the driving process.
基金supported by the National Science Foundation(2343619,2416872,2244219,and 2146497).
文摘In recent research on the Digital Twin-based Vehicular Ad hoc Network(DT-VANET),Federated Learning(FL)has shown its ability to provide data privacy.However,Federated learning struggles to adequately train a global model when confronted with data heterogeneity and data sparsity among vehicles,which ensure suboptimal accuracy in making precise predictions for different vehicle types.To address these challenges,this paper combines Federated Transfer Learning(FTL)to conduct vehicle clustering related to types of vehicles and proposes a novel Hierarchical Federated Transfer Learning(HFTL).We construct a framework for DT-VANET,along with two algorithms designed for cloud server model updates and intra-cluster federated transfer learning,to improve the accuracy of the global model.In addition,we developed a data quality score-based mechanism to prevent the global model from being affected by malicious vehicles.Lastly,detailed experiments on real-world datasets are conducted,considering different performance metrics that verify the effectiveness and efficiency of our algorithm.
基金The authors would like to appreciate the financial support of the National Natural Science Foundation of China(Grant No.61703041)the technological innovation program of Beijing Institute of Technology(2021CX11006).
文摘As intelligent vehicles usually have complex overtaking process,a safe and efficient automated overtaking system(AOS)is vital to avoid accidents caused by wrong operation of drivers.Existing AOSs rarely consider longitudinal reactions of the overtaken vehicle(OV)during overtaking.This paper proposed a novel AOS based on hierarchical reinforcement learning,where the longitudinal reaction is given by a data-driven social preference estimation.This AOS incorporates two modules that can function in different overtaking phases.The first module based on semi-Markov decision process and motion primitives is built for motion planning and control.The second module based on Markov decision process is designed to enable vehicles to make proper decisions according to the social preference of OV.Based on realistic overtaking data,the proposed AOS and its modules are verified experimentally.The results of the tests show that the proposed AOS can realize safe and effective overtaking in scenes built by realistic data,and has the ability to flexibly adjust lateral driving behavior and lane changing position when the OVs have different social preferences.
基金supported by the National Basic Research Program of China (2013CB329603)the National Natural Science Foundation of China (61375058, 71231002)+1 种基金the China Mobile Research Fund (MCM 20130351)the Ministry of Education of China and the Special Co-Construction Project of Beijing Municipal Commission of Education
文摘Option is a promising method to discover the hierarchical structure in reinforcement learning （RL） for learning acceleration. The key to option discovery is about how an agent can find useful subgoals autonomically among the passing trails. By analyzing the agent＇s actions in the trails, useful heuristics can be found. Not only does the agent pass subgoals more frequently, but also its effective actions are restricted in subgoals. As a consequence, the subgoals can be deemed as the most matching action-restricted states in the paths. In the grid-world environment, the concept of the unique-direction value reflecting the action-restricted property was introduced to find the most matching action-restricted states. The unique-direction-value （UDV） approach is chosen to form options offline and online autonomically. Experiments show that the approach can find subgoals correctly. Thus the Q-learning with options found on both offline and online process can accelerate learning significantly.
基金Project supported by the National Natural Science Foundation of China(No.61379074)the Zhejiang Provincial Natural Science Foundation of China(Nos.LZ12F02003 and LY15F020035)
文摘In some image classification tasks, similarities among different categories are different and the samples are usually misclassified as highly similar categories. To distinguish highly similar categories, more specific features are required so that the classifier can improve the classification performance. In this paper, we propose a novel two-level hierarchical feature learning framework based on the deep convolutional neural network(CNN), which is simple and effective. First, the deep feature extractors of different levels are trained using the transfer learning method that fine-tunes the pre-trained deep CNN model toward the new target dataset. Second, the general feature extracted from all the categories and the specific feature extracted from highly similar categories are fused into a feature vector. Then the final feature representation is fed into a linear classifier. Finally, experiments using the Caltech-256, Oxford Flower-102, and Tasmania Coral Point Count(CPC) datasets demonstrate that the expression ability of the deep features resulting from two-level hierarchical feature learning is powerful. Our proposed method effectively increases the classification accuracy in comparison with flat multiple classification methods.
基金financial support provided by the Major Project of Yunnan Science and Technology,under Project No.202302AE09002003,entitled“Research on the Integration of Key Technologies in Smart Agriculture.”。
文摘This study proposed a weighted sampling hierarchical classification learning method based on an efficient backbone network model to address the problems of high costs,low accuracy,and time-consuming traditional tea disease recognition methods.This method enhances the feature extraction ability by conducting hierarchical classification learning based on the EfficientNet model,effectively alleviating the impact of high similarity between tea diseases on the model’s classification performance.To better solve the problem of few and unevenly distributed tea disease samples,this study introduced a weighted sampling scheme to optimize data processing,which not only alleviates the overfitting effect caused by too few sample data but also balances the probability of extracting imbalanced classification data.The experimental results show that the proposed method was significant in identifying both healthy tea leaves and four common leaf diseases of tea(tea algal spot disease,tea white spot disease,tea anthracnose disease,and tea leaf blight disease).After applying the“weighted sampling hierarchical classification learning method”to train 7 different efficient backbone networks,most of their accuracies have improved.The EfficientNet-B1 model proposed in this study achieved an accuracy rate of 99.21%after adopting this learning method,which is higher than EfficientNet-b2(98.82%)and MobileNet-V3(98.43%).In addition,to better apply the results of identifying tea diseases,this study developed a mini-program that operates on WeChat.Users can quickly obtain accurate identification results and corresponding disease descriptions and prevention methods through simple operations.This intelligent tool for identifying tea diseases can serve as an auxiliary tool for farmers,consumers,and related scientific researchers and has certain practical value.
基金supported by the National Natural Science Foundation of China(Nos.22178383 and 21706282)Beijing Natural Science Foundation(No.2232021)Research Foundation of China University of Petroleum(Beijing)(No.2462020BJRC004).
文摘Shuttle tankers scheduling is an important task in offshore oil and gas transportation process,which involves operating time window fulfillment,optimal transportation planning,and proper inventory management.However,conventional approaches like Mixed lnteger Linear Programming(MlLP)or meta heuristic algorithms often fail in long running time.In this paper,a Graph Pointer Network(GPN)based Hierarchical Curriculum Reinforcement Learning(HCRl)method is proposed to solve Shuttle Tankers Scheduling Problem(STSP)The model is trained to divide STSP into voyage and operation stages and generate routing and inventory management decisions sequentially.An asynchronous training strategy is developed to address the coupling between stages.Comparison experiments demonstrate that the proposed HCRL method achieves 12%shortel tour lengths on average compared to heuristic algorithms.Additional experiments validate its generalizability to unseen instances and scalability to larger instances.
