The quality of designed structures embedded in rocks is strongly related to rock strength parameters of intact rock.Measuring different parameters from tests could be very expensive in designing phase of projects.Esti...The quality of designed structures embedded in rocks is strongly related to rock strength parameters of intact rock.Measuring different parameters from tests could be very expensive in designing phase of projects.Estimating some parameters from other ones can reduce costs and time of project procedure.In this paper,the relationships between static and dynamic parameters of marls are studied by using the single and multiple linear regressions.For this purpose,several marl core samples from Seydoon region,Khoozestan Province in Iran are collected and tested.Some equations with sufficient correlation have been obtained to predict the engineering parameters of marls,especially the uniaxial compressive strength(UCS).展开更多
Hemipelagic to pelagic(H/P)marls,representing pelitic deposits,accumulated within the foredeep sub-basin of the Dinaric Foreland Basin(northern Neotethyan margin,present-day Croatia)during the Middle to Late Eocene.Sy...Hemipelagic to pelagic(H/P)marls,representing pelitic deposits,accumulated within the foredeep sub-basin of the Dinaric Foreland Basin(northern Neotethyan margin,present-day Croatia)during the Middle to Late Eocene.Syn-sedimentary tectonic movements,paleogeographic position and exchanges of short-lived hyperthermal episodes affected the sedimentation and related mineral and geochemical record of these deposits.Mineral(clay)assemblages bear signature of prevailing physical weathering with significant illite and chlorite content,but climatic seasonality is suggested by smectite-interlayered phases and sporadical increase of kaolinite content.Illite crystallinity varies significantly,and the lowest crystallinity is recorded by the Lutetian samples.Illite chemistry index is always bellow 0.5,being characteristic for Fe-Mg-rich illite.The geochemical records are the most prominent(CIA up to 76,CIW up to 91)for the Istrian Lutetian(42.3-40.5 Ma),but also for Priabonian(35.8-34.3 Ma)samples of Hvar Island.The ICV values(the lowest 1.40 and the highest 10.85)of all studied samples fall above PAAS(ICV=0.85)and point to their chemical immaturity.The Ga/Rb ratios are lower than 0.2 and K_(2)O/Al_(2)O_(3) ratios are also low(0.16-0.22),implying transition between cold and dry,and warm and humid climate,obviously trending among several warming episodes.展开更多
This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary obj...This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary objective is to explore the unknown environments to locate and track targets effectively. To address this problem, we propose a novel Multi-Agent Reinforcement Learning (MARL) method based on Graph Neural Network (GNN). Firstly, a method is introduced for encoding continuous-space multi-UAV problem data into spatial graphs which establish essential relationships among agents, obstacles, and targets. Secondly, a Graph AttenTion network (GAT) model is presented, which focuses exclusively on adjacent nodes, learns attention weights adaptively and allows agents to better process information in dynamic environments. Reward functions are specifically designed to tackle exploration challenges in environments with sparse rewards. By introducing a framework that integrates centralized training and distributed execution, the advancement of models is facilitated. Simulation results show that the proposed method outperforms the existing MARL method in search rate and tracking performance with less collisions. The experiments show that the proposed method can be extended to applications with a larger number of agents, which provides a potential solution to the challenging problem of multi-UAV autonomous tracking in dynamic unknown environments.展开更多
Marine organic-rich marl is not only a high-quality hydrocarbon source of conventional oil and gas,but also a new type and field of unconventional oil and gas exploration.An understanding of its pore structure evoluti...Marine organic-rich marl is not only a high-quality hydrocarbon source of conventional oil and gas,but also a new type and field of unconventional oil and gas exploration.An understanding of its pore structure evolution characteristics during a hydrocarbon generation process is theoretically significant and has application prospects for the exploration and development of this special type of natural gas reservoirs.This study conducted thermal simulation of hydrocarbon generation under near-geological conditions during a whole process for cylinder samples of low mature marine organic-rich marl in the Middle Devonian of Luquan,Yunnan Province,China.During this process,hydrocarbon products at different evolution stages were quantified and corresponding geochemical properties were analyzed.Simultaneously,field emission scanning electron microscopy(FE-SEM)and low-pressure gas adsorption(CO_(2),N_(2))tests were applied to the corresponding cylinder residue samples to reveal the mechanisms of different types of pore formation and evolution,and clarify the dynamic evolution processes of their pore systems.The results show that with an increase in temperature and pressure,the total oil yield peaks at an equivalent vitrinite reflectance(VR_(o))of 1.03%and is at the maximum retention stage of liquid hydrocarbons,which are 367.51 mg/g TOC and 211.67 mg/g TOC,respectively.The hydrocarbon gas yield increases continuously with an increase in maturity.The high retained oil rate at the peak of oil generation provides an abundant material basis for gas formation at high maturity and over-maturity stage.The lower limit of VR_(o)for organic matter(OM)pore mass development is about 1.6%,and bitumen pores,organic-clay complex pores together with intergranular pores,grain edge seams and dissolution pores constitute a complicated pore-seam-network system,which is the main reservoir space for unconventional carbonate gas.Pore formation and evolution are controlled synergistically by hydrocarbon generation,diagenesis and organic-inorganic interactions,and the pattern of pore structure evolution can be divided into four stages.A pore volume(PV)and a specific surface area(SSA)are at their highest values within the maturity range of 1.9%to 2.5%,which is conducive to exploring unconventional natural gas.展开更多
针对移动边缘计算中用户移动性导致服务器间负载分布不均,用户服务质量(Quality of Service,QoS)下降的问题,提出了一种移动性感知下的分布式任务迁移方案。首先,以优化网络中性能最差的用户QoS为目标,建立了一个长期极大极小化公平性问...针对移动边缘计算中用户移动性导致服务器间负载分布不均,用户服务质量(Quality of Service,QoS)下降的问题,提出了一种移动性感知下的分布式任务迁移方案。首先,以优化网络中性能最差的用户QoS为目标,建立了一个长期极大极小化公平性问题(Max Min Fairness,MMF),利用李雅普诺夫(Lyapunov)优化将原问题转化解耦。然后,将其建模为去中心化部分可观测马尔可夫决策过程(Decentralized Partially Observable Markov Decision Process,Dec-POMDP),提出一种基于多智能体柔性演员-评论家(Soft Actor-Critic,SAC)的分布式任务迁移算法,将奖励函数解耦为节点奖励和用户个体奖励,分别基于节点负载均衡度和用户QoS施加奖励。仿真结果表明,相比于现有任务迁移方案,所提算法能够在保证用户QoS的前提下降低任务迁移率,保证系统负载均衡。展开更多
Highly intelligent Unmanned Combat Aerial Vehicle(UCAV)formation is expected to bring out strengths in Beyond-Visual-Range(BVR)air combat.Although Multi-Agent Reinforcement Learning(MARL)shows outstanding performance ...Highly intelligent Unmanned Combat Aerial Vehicle(UCAV)formation is expected to bring out strengths in Beyond-Visual-Range(BVR)air combat.Although Multi-Agent Reinforcement Learning(MARL)shows outstanding performance in cooperative decision-making,it is challenging for existing MARL algorithms to quickly converge to an optimal strategy for UCAV formation in BVR air combat where confrontation is complicated and reward is extremely sparse and delayed.Aiming to solve this problem,this paper proposes an Advantage Highlight Multi-Agent Proximal Policy Optimization(AHMAPPO)algorithm.First,at every step,the AHMAPPO records the degree to which the best formation exceeds the average of formations in parallel environments and carries out additional advantage sampling according to it.Then,the sampling result is introduced into the updating process of the actor network to improve its optimization efficiency.Finally,the simulation results reveal that compared with some state-of-the-art MARL algorithms,the AHMAPPO can obtain a more excellent strategy utilizing fewer sample episodes in the UCAV formation BVR air combat simulation environment built in this paper,which can reflect the critical features of BVR air combat.The AHMAPPO can significantly increase the convergence efficiency of the strategy for UCAV formation in BVR air combat,with a maximum increase of 81.5%relative to other algorithms.展开更多
Municipal solid waste(MSW)is accumulating over elapsed time across the world,and it is observed in many projects associated with weak soils,such as marl.Therefore,effective solutions to the environmental problem are e...Municipal solid waste(MSW)is accumulating over elapsed time across the world,and it is observed in many projects associated with weak soils,such as marl.Therefore,effective solutions to the environmental problem are essential.Conventional techniques for stabilizing marl generally use substances such as lime and cement,which could exacerbate pollution.For this,some new stabilizers,e.g.nano-MgO,are used.There are large quantities of marls and MSW in Shiraz City,Iran.The present study aims to evaluate the feasibility of using nano-MgO as a green low-carbon binder to remove MSW from the environment and make construction projects more cost-effective.Consolidated drained shear tests were conducted to evaluate the mechanical behaviors of the nano-MgO treated marl specimens at high normal stresses.The marl specimens containing MSW percentages of 15%,25%,35%,and 45%and nano-MgO percentages of 0.25%,0.5%,0.75%,and 1%,were used.It is found that the marl containing 15%and 25%MSW and 0.5%nano-MgO at 28-d curing can perform cation exchange and form new cementitious products.The soils with merely MSW show good performance due to the removal of the kaolinite and the formation of brucite.However,the soil with 25%MSW and 0.5%nano-MgO shows the same strength enhancement as the specimen with the optimal nano-MgO(0.75%)through the formation of dolomite,with a 20.59%increase in strain energy(SE).展开更多
Efficient exploration in complex coordination tasks has been considered a challenging problem in multi-agent reinforcement learning(MARL). It is significantly more difficult for those tasks with latent variables that ...Efficient exploration in complex coordination tasks has been considered a challenging problem in multi-agent reinforcement learning(MARL). It is significantly more difficult for those tasks with latent variables that agents cannot directly observe. However, most of the existing latent variable discovery methods lack a clear representation of latent variables and an effective evaluation of the influence of latent variables on the agent. In this paper, we propose a new MARL algorithm based on the soft actor-critic method for complex continuous control tasks with confounders. It is called the multi-agent soft actor-critic with latent variable(MASAC-LV) algorithm, which uses variational inference theory to infer the compact latent variables representation space from a large amount of offline experience.Besides, we derive the counterfactual policy whose input has no latent variables and quantify the difference between the actual policy and the counterfactual policy via a distance function. This quantified difference is considered an intrinsic motivation that gives additional rewards based on how much the latent variable affects each agent. The proposed algorithm is evaluated on two collaboration tasks with confounders, and the experimental results demonstrate the effectiveness of MASAC-LV compared to other baseline algorithms.展开更多
This paper examines the difficulties of managing distributed power systems,notably due to the increasing use of renewable energy sources,and focuses on voltage control challenges exacerbated by their variable nature i...This paper examines the difficulties of managing distributed power systems,notably due to the increasing use of renewable energy sources,and focuses on voltage control challenges exacerbated by their variable nature in modern power grids.To tackle the unique challenges of voltage control in distributed renewable energy networks,researchers are increasingly turning towards multi-agent reinforcement learning(MARL).However,MARL raises safety concerns due to the unpredictability in agent actions during their exploration phase.This unpredictability can lead to unsafe control measures.To mitigate these safety concerns in MARL-based voltage control,our study introduces a novel approach:Safety-ConstrainedMulti-Agent Reinforcement Learning(SC-MARL).This approach incorporates a specialized safety constraint module specifically designed for voltage control within the MARL framework.This module ensures that the MARL agents carry out voltage control actions safely.The experiments demonstrate that,in the 33-buses,141-buses,and 322-buses power systems,employing SC-MARL for voltage control resulted in a reduction of the Voltage Out of Control Rate(%V.out)from0.43,0.24,and 2.95 to 0,0.01,and 0.03,respectively.Additionally,the Reactive Power Loss(Q loss)decreased from 0.095,0.547,and 0.017 to 0.062,0.452,and 0.016 in the corresponding systems.展开更多
Multi-Agent Reinforcement Learning(MARL)has proven to be successful in cooperative assignments.MARL is used to investigate how autonomous agents with the same interests can connect and act in one team.MARL cooperation...Multi-Agent Reinforcement Learning(MARL)has proven to be successful in cooperative assignments.MARL is used to investigate how autonomous agents with the same interests can connect and act in one team.MARL cooperation scenarios are explored in recreational cooperative augmented reality environments,as well as realworld scenarios in robotics.In this paper,we explore the realm of MARL and its potential applications in cooperative assignments.Our focus is on developing a multi-agent system that can collaborate to attack or defend against enemies and achieve victory withminimal damage.To accomplish this,we utilize the StarCraftMulti-Agent Challenge(SMAC)environment and train four MARL algorithms:Q-learning with Mixtures of Experts(QMIX),Value-DecompositionNetwork(VDN),Multi-agent Proximal PolicyOptimizer(MAPPO),andMulti-Agent Actor Attention Critic(MAA2C).These algorithms allow multiple agents to cooperate in a specific scenario to achieve the targeted mission.Our results show that the QMIX algorithm outperforms the other three algorithms in the attacking scenario,while the VDN algorithm achieves the best results in the defending scenario.Specifically,the VDNalgorithmreaches the highest value of battle wonmean and the lowest value of dead alliesmean.Our research demonstrates the potential forMARL algorithms to be used in real-world applications,such as controllingmultiple robots to provide helpful services or coordinating teams of agents to accomplish tasks that would be impossible for a human to do.The SMAC environment provides a unique opportunity to test and evaluate MARL algorithms in a challenging and dynamic environment,and our results show that these algorithms can be used to achieve victory with minimal damage.展开更多
The disorders caused by the swelling of the soil on the structures have been observed for several years in the city of Rufisque. This article presents the results of the study of swelling kinetics of expansive soils i...The disorders caused by the swelling of the soil on the structures have been observed for several years in the city of Rufisque. This article presents the results of the study of swelling kinetics of expansive soils in Rufisque and their prediction based on the hyperbolic rule. The odometer is used as an instrument for measuring swelling and the tests are carried out on some intact samples at their sampling water content. The present study shows that in Rufisque the most swelling layer is marl. The results show two phases of development. The first phase is very fast and represents 77% of the final deformation and the second one is slower. The prediction of the issue by the hyperbolic rule shows that it underestimates the first phase but gives a good prediction of the second phase of the swelling rate. There is a good correlation between the final swelling rates. However, the model gives a bad approximation of the half-swelling time.展开更多
Leakage of oil and its derivatives into the soil can change the engineering behavior of soil as well as cause environmental disasters.Also,recovering the contaminated sites into their natural condition and making cont...Leakage of oil and its derivatives into the soil can change the engineering behavior of soil as well as cause environmental disasters.Also,recovering the contaminated sites into their natural condition and making contaminated materials as both environmentally and geotechnically suitable construction materials need the employment of remediation techniques.Bioremediation,as an efficient,low cost and environmentalfriendly approach,was used in the case of highly plastic clayey soils.To better understand the change in geotechnical properties of highly plastic fine-grained soil due to crude oil contamination and bioremediation,Atterberg limits,compaction,unconfined compression,direct shear,and consolidation tests were conducted on natural,contaminated,and bioremediated soil samples to investigate the effects of contamination and remediation on fine-grained soil properties.Oil contamination reduced maximum dry density(MDD),optimum moisture content(OMC),unconfined compressive strength(UCS),shear strength,swelling pressure,and coefficient of consolidation of soil.In addition,contamination increased the compression and swelling indices and compressibility of soil.Bioremediation reduced soil contamination by about 50%.Moreover,in comparison with contaminated soil,bioremediation reduced the MDD,UCS,swelling index,free swelling and swelling pressure of soil,and also increased OMC,shear strength,cohesion,internal friction angle,failure strain,porosity,compression index,and settlement.Microstructural analyses showed that oil contamination does not alter the soil structure in terms of chemical compounds,elements,and constituent minerals.While it decreased the specific surface area of the soil,and the bioremediation significantly increased the mentioned parameters.Bioremediation resulted in the formation of quasi-fibrous textures and porous and agglomerated structures.As a result,oil contamination affected the mechanical properties of soil negatively,but bioremediation improved these properties.展开更多
Due to the complicated lithology in the ES3 Member of the Shahejie Formation in the Shulu sag,Jizhong depression,it is difficult to classify the rock types and characterize the reservoirs at the marl intervals.In this...Due to the complicated lithology in the ES3 Member of the Shahejie Formation in the Shulu sag,Jizhong depression,it is difficult to classify the rock types and characterize the reservoirs at the marl intervals.In this paper,a four-element classification method has been proposed,and seven rock types have been identified by analyzing the mineral composition.The primary rock types are medium-high organic carbonate rocks and medium-high organic shaly-siliceous carbonate rocks.With the methods of field emission scanning electron microscopy,high-pressure mercury intrusion,nitrogen adsorption,and nano-CT,four types of reservoir spaces have been identified,including intra-granular pores,intergranular pores(inter-crystalline pores),organic pores,and micro-fractures.By combining the method of high-pressure mercury intrusion with the method of the nitrogen adsorption,the porosity of the marl has been measured,ranging from 0.73%to 5.39%.The distribution of the pore sizes is bimodal,and the pore types are dominated by micron pores.Through this study,it has been concluded that the sag area to the east of Well ST1H is the favorable area for the development of self-sourced and self-reservoired shale oil.According to the results of geochemical and reservoir analysis,the III Oil Group may have sweet spot layers.展开更多
In this paper, the reinforcement learning method for cooperative multi-agent systems(MAS) with incremental number of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with...In this paper, the reinforcement learning method for cooperative multi-agent systems(MAS) with incremental number of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with a specific number of agents, and can learn well-performed policies. However, if there is an increasing number of agents, the previously learned in may not perform well in the current scenario. The new agents need to learn from scratch to find optimal policies with others,which may slow down the learning speed of the whole team. To solve that problem, in this paper, we propose a new algorithm to take full advantage of the historical knowledge which was learned before, and transfer it from the previous agents to the new agents. Since the previous agents have been trained well in the source environment, they are treated as teacher agents in the target environment. Correspondingly, the new agents are called student agents. To enable the student agents to learn from the teacher agents, we first modify the input nodes of the networks for teacher agents to adapt to the current environment. Then, the teacher agents take the observations of the student agents as input, and output the advised actions and values as supervising information. Finally, the student agents combine the reward from the environment and the supervising information from the teacher agents, and learn the optimal policies with modified loss functions. By taking full advantage of the knowledge of teacher agents, the search space for the student agents will be reduced significantly, which can accelerate the learning speed of the holistic system. The proposed algorithm is verified in some multi-agent simulation environments, and its efficiency has been demonstrated by the experiment results.展开更多
In 2022,the risk exploration well Chongtan1(CT1)in the Sichuan Basin revealed commercial oil and gas flow during test in a new zone–the marl of the second submember of the third member of Leikoupo Formation(Lei-32)of...In 2022,the risk exploration well Chongtan1(CT1)in the Sichuan Basin revealed commercial oil and gas flow during test in a new zone–the marl of the second submember of the third member of Leikoupo Formation(Lei-32)of Middle Triassic,recording a significant discovery.However,the hydrocarbon accumulation in marl remains unclear,which restricts the selection and deployment of exploration area.Focusing on Well CT1,the hydrocarbon accumulation characteristics of Lei-32 marl are analyzed to clarify the potential zones for exploration.The following findings are obtained.First,according to the geochemical analysis of petroleum and source rocks,oil and gas in the Lei-32 marl of Well CT1 are originated from the same marl.The marl acts as both source rock and reservoir rock.Second,the Lei-32 marl in central Sichuan Basin is of lagoonal facies,with a thickness of 40–130 m,an area of about 40000 km^(2),a hydrocarbon generation intensity of(4–12)×10^(8) m^(3)/km^(2),and an estimated quantity of generated hydrocarbons of 25×10^(12) m^(3).Third,the lagoonal marl reservoirs are widely distributed in central Sichuan Basin.Typically,in Xichong–Yilong,Ziyang–Jianyang and Moxi South,the reservoirs are 20–60 m thick and cover an area of 7500 km^(2).Fourth,hydrocarbons in the lagoonal marl are generated and stored in the Lei-32 marl,which means that marl serves as both source rock and reservoir rock.They represent a new type of unconventional resource,which is worthy of exploring.Fifth,based on the interpretation of 2D and 3D seismic data from central Sichuan Basin,Xichong and Suining are defined as favorable prospects with estimated resources of(2000–3000)×10^(8) m^(3).展开更多
In fractured reservoirs characterized by low matrix permeability,fracture networks control the main fluid flow paths.However,in layered reservoirs,the vertical extension of fractures is often restricted to single laye...In fractured reservoirs characterized by low matrix permeability,fracture networks control the main fluid flow paths.However,in layered reservoirs,the vertical extension of fractures is often restricted to single layers.In this study,we explored the effect of changing marl/shale thickness on fracture extension using comprehensive field data and numerical modeling.The field data were sampled from coastal exposures of Liassic limestone-marl/shale alternations in Wales and Somerset(Bristol Channel Basin,UK).The vertical fracture traces of more than 4000 fractures were mapped in detail.Six sections were selected to represent a variety of layer thicknesses.Besides the field data also thin sections were analyzed.Numerical models of fracture extension in a two-layer limestone-marl system were based on field data and laboratory measurements of Young's moduli.The modeled principal stress magnitude σ3 along the lithological contact was used as an indication for fracture extension through marls.Field data exhibit good correlation(R^2=0.76) between fracture extension and marl thickness,the thicker the marl layer the fewer fractures propagate through.The model results show that almost no tensile stress reaches the top of the marl layer when the marls are thicker than 30 cm.For marls that are less than 20 cm,the propagation of stress is more dependent on the stiffness of the marls.The higher the contrast between limestone and marl stiffness the lower the stress that is transmitted into the marl layer.In both model experiments and field data the critical marl thickness for fracture extension is ca.15-20 cm.This quantification of critical marl thicknesses can be used to improve predictions of fracture networks and permeability in layered rocks.Up-or downsampling methods often ignore spatially continuous impermeable layers with thicknesses that are under the detection limit of seismic data.However,ignoring these layers can lead to overestimates of the overall permeability.Therefore,the understanding of how fractures propagate and terminate through impermeable layers will help to improve the characterization of conventional reservoirs.展开更多
文摘The quality of designed structures embedded in rocks is strongly related to rock strength parameters of intact rock.Measuring different parameters from tests could be very expensive in designing phase of projects.Estimating some parameters from other ones can reduce costs and time of project procedure.In this paper,the relationships between static and dynamic parameters of marls are studied by using the single and multiple linear regressions.For this purpose,several marl core samples from Seydoon region,Khoozestan Province in Iran are collected and tested.Some equations with sufficient correlation have been obtained to predict the engineering parameters of marls,especially the uniaxial compressive strength(UCS).
基金supported by Croatian Science Foundation Research Project Dinaridic Foreland Basin between Two Eocene Thermal Optima:A Possible Scenario for the Northern Adriatic BREEMECO(No.2019-04-5775)。
文摘Hemipelagic to pelagic(H/P)marls,representing pelitic deposits,accumulated within the foredeep sub-basin of the Dinaric Foreland Basin(northern Neotethyan margin,present-day Croatia)during the Middle to Late Eocene.Syn-sedimentary tectonic movements,paleogeographic position and exchanges of short-lived hyperthermal episodes affected the sedimentation and related mineral and geochemical record of these deposits.Mineral(clay)assemblages bear signature of prevailing physical weathering with significant illite and chlorite content,but climatic seasonality is suggested by smectite-interlayered phases and sporadical increase of kaolinite content.Illite crystallinity varies significantly,and the lowest crystallinity is recorded by the Lutetian samples.Illite chemistry index is always bellow 0.5,being characteristic for Fe-Mg-rich illite.The geochemical records are the most prominent(CIA up to 76,CIW up to 91)for the Istrian Lutetian(42.3-40.5 Ma),but also for Priabonian(35.8-34.3 Ma)samples of Hvar Island.The ICV values(the lowest 1.40 and the highest 10.85)of all studied samples fall above PAAS(ICV=0.85)and point to their chemical immaturity.The Ga/Rb ratios are lower than 0.2 and K_(2)O/Al_(2)O_(3) ratios are also low(0.16-0.22),implying transition between cold and dry,and warm and humid climate,obviously trending among several warming episodes.
基金supported by the National Natural Science Foundation of China(Nos.12272104,U22B2013).
文摘This paper investigates the challenges associated with Unmanned Aerial Vehicle (UAV) collaborative search and target tracking in dynamic and unknown environments characterized by limited field of view. The primary objective is to explore the unknown environments to locate and track targets effectively. To address this problem, we propose a novel Multi-Agent Reinforcement Learning (MARL) method based on Graph Neural Network (GNN). Firstly, a method is introduced for encoding continuous-space multi-UAV problem data into spatial graphs which establish essential relationships among agents, obstacles, and targets. Secondly, a Graph AttenTion network (GAT) model is presented, which focuses exclusively on adjacent nodes, learns attention weights adaptively and allows agents to better process information in dynamic environments. Reward functions are specifically designed to tackle exploration challenges in environments with sparse rewards. By introducing a framework that integrates centralized training and distributed execution, the advancement of models is facilitated. Simulation results show that the proposed method outperforms the existing MARL method in search rate and tracking performance with less collisions. The experiments show that the proposed method can be extended to applications with a larger number of agents, which provides a potential solution to the challenging problem of multi-UAV autonomous tracking in dynamic unknown environments.
基金supported by the National Natural Science Foundation of China(Grant No.41930426)。
文摘Marine organic-rich marl is not only a high-quality hydrocarbon source of conventional oil and gas,but also a new type and field of unconventional oil and gas exploration.An understanding of its pore structure evolution characteristics during a hydrocarbon generation process is theoretically significant and has application prospects for the exploration and development of this special type of natural gas reservoirs.This study conducted thermal simulation of hydrocarbon generation under near-geological conditions during a whole process for cylinder samples of low mature marine organic-rich marl in the Middle Devonian of Luquan,Yunnan Province,China.During this process,hydrocarbon products at different evolution stages were quantified and corresponding geochemical properties were analyzed.Simultaneously,field emission scanning electron microscopy(FE-SEM)and low-pressure gas adsorption(CO_(2),N_(2))tests were applied to the corresponding cylinder residue samples to reveal the mechanisms of different types of pore formation and evolution,and clarify the dynamic evolution processes of their pore systems.The results show that with an increase in temperature and pressure,the total oil yield peaks at an equivalent vitrinite reflectance(VR_(o))of 1.03%and is at the maximum retention stage of liquid hydrocarbons,which are 367.51 mg/g TOC and 211.67 mg/g TOC,respectively.The hydrocarbon gas yield increases continuously with an increase in maturity.The high retained oil rate at the peak of oil generation provides an abundant material basis for gas formation at high maturity and over-maturity stage.The lower limit of VR_(o)for organic matter(OM)pore mass development is about 1.6%,and bitumen pores,organic-clay complex pores together with intergranular pores,grain edge seams and dissolution pores constitute a complicated pore-seam-network system,which is the main reservoir space for unconventional carbonate gas.Pore formation and evolution are controlled synergistically by hydrocarbon generation,diagenesis and organic-inorganic interactions,and the pattern of pore structure evolution can be divided into four stages.A pore volume(PV)and a specific surface area(SSA)are at their highest values within the maturity range of 1.9%to 2.5%,which is conducive to exploring unconventional natural gas.
基金co-supported by the National Natural Science Foundation of China(No.52272382)the Aeronautical Science Foundation of China(No.20200017051001)the Fundamental Research Funds for the Central Universities,China.
文摘Highly intelligent Unmanned Combat Aerial Vehicle(UCAV)formation is expected to bring out strengths in Beyond-Visual-Range(BVR)air combat.Although Multi-Agent Reinforcement Learning(MARL)shows outstanding performance in cooperative decision-making,it is challenging for existing MARL algorithms to quickly converge to an optimal strategy for UCAV formation in BVR air combat where confrontation is complicated and reward is extremely sparse and delayed.Aiming to solve this problem,this paper proposes an Advantage Highlight Multi-Agent Proximal Policy Optimization(AHMAPPO)algorithm.First,at every step,the AHMAPPO records the degree to which the best formation exceeds the average of formations in parallel environments and carries out additional advantage sampling according to it.Then,the sampling result is introduced into the updating process of the actor network to improve its optimization efficiency.Finally,the simulation results reveal that compared with some state-of-the-art MARL algorithms,the AHMAPPO can obtain a more excellent strategy utilizing fewer sample episodes in the UCAV formation BVR air combat simulation environment built in this paper,which can reflect the critical features of BVR air combat.The AHMAPPO can significantly increase the convergence efficiency of the strategy for UCAV formation in BVR air combat,with a maximum increase of 81.5%relative to other algorithms.
文摘Municipal solid waste(MSW)is accumulating over elapsed time across the world,and it is observed in many projects associated with weak soils,such as marl.Therefore,effective solutions to the environmental problem are essential.Conventional techniques for stabilizing marl generally use substances such as lime and cement,which could exacerbate pollution.For this,some new stabilizers,e.g.nano-MgO,are used.There are large quantities of marls and MSW in Shiraz City,Iran.The present study aims to evaluate the feasibility of using nano-MgO as a green low-carbon binder to remove MSW from the environment and make construction projects more cost-effective.Consolidated drained shear tests were conducted to evaluate the mechanical behaviors of the nano-MgO treated marl specimens at high normal stresses.The marl specimens containing MSW percentages of 15%,25%,35%,and 45%and nano-MgO percentages of 0.25%,0.5%,0.75%,and 1%,were used.It is found that the marl containing 15%and 25%MSW and 0.5%nano-MgO at 28-d curing can perform cation exchange and form new cementitious products.The soils with merely MSW show good performance due to the removal of the kaolinite and the formation of brucite.However,the soil with 25%MSW and 0.5%nano-MgO shows the same strength enhancement as the specimen with the optimal nano-MgO(0.75%)through the formation of dolomite,with a 20.59%increase in strain energy(SE).
基金supported in part by the National Natural Science Foundation of China (62136008,62236002,61921004,62173251,62103104)the “Zhishan” Scholars Programs of Southeast Universitythe Fundamental Research Funds for the Central Universities (2242023K30034)。
文摘Efficient exploration in complex coordination tasks has been considered a challenging problem in multi-agent reinforcement learning(MARL). It is significantly more difficult for those tasks with latent variables that agents cannot directly observe. However, most of the existing latent variable discovery methods lack a clear representation of latent variables and an effective evaluation of the influence of latent variables on the agent. In this paper, we propose a new MARL algorithm based on the soft actor-critic method for complex continuous control tasks with confounders. It is called the multi-agent soft actor-critic with latent variable(MASAC-LV) algorithm, which uses variational inference theory to infer the compact latent variables representation space from a large amount of offline experience.Besides, we derive the counterfactual policy whose input has no latent variables and quantify the difference between the actual policy and the counterfactual policy via a distance function. This quantified difference is considered an intrinsic motivation that gives additional rewards based on how much the latent variable affects each agent. The proposed algorithm is evaluated on two collaboration tasks with confounders, and the experimental results demonstrate the effectiveness of MASAC-LV compared to other baseline algorithms.
基金“Regional Innovation Strategy(RIS)”through the National Research Foundation of Korea(NRF)funded by the Ministry of Education(MOE)(2021RIS-002).
文摘This paper examines the difficulties of managing distributed power systems,notably due to the increasing use of renewable energy sources,and focuses on voltage control challenges exacerbated by their variable nature in modern power grids.To tackle the unique challenges of voltage control in distributed renewable energy networks,researchers are increasingly turning towards multi-agent reinforcement learning(MARL).However,MARL raises safety concerns due to the unpredictability in agent actions during their exploration phase.This unpredictability can lead to unsafe control measures.To mitigate these safety concerns in MARL-based voltage control,our study introduces a novel approach:Safety-ConstrainedMulti-Agent Reinforcement Learning(SC-MARL).This approach incorporates a specialized safety constraint module specifically designed for voltage control within the MARL framework.This module ensures that the MARL agents carry out voltage control actions safely.The experiments demonstrate that,in the 33-buses,141-buses,and 322-buses power systems,employing SC-MARL for voltage control resulted in a reduction of the Voltage Out of Control Rate(%V.out)from0.43,0.24,and 2.95 to 0,0.01,and 0.03,respectively.Additionally,the Reactive Power Loss(Q loss)decreased from 0.095,0.547,and 0.017 to 0.062,0.452,and 0.016 in the corresponding systems.
基金supported in part by United States Air Force Research Institute for Tactical Autonomy(RITA)University Affiliated Research Center(UARC)in part by the United States Air Force Office of Scientific Research(AFOSR)Contract FA9550-22-1-0268 awarded to KHA,https://www.afrl.af.mil/AFOSR/The contract is entitled:“Investigating Improving Safety of Autonomous Exploring Intelligent Agents with Human-in-the-Loop Reinforcement Learning,”and in part by Jackson State University.
文摘Multi-Agent Reinforcement Learning(MARL)has proven to be successful in cooperative assignments.MARL is used to investigate how autonomous agents with the same interests can connect and act in one team.MARL cooperation scenarios are explored in recreational cooperative augmented reality environments,as well as realworld scenarios in robotics.In this paper,we explore the realm of MARL and its potential applications in cooperative assignments.Our focus is on developing a multi-agent system that can collaborate to attack or defend against enemies and achieve victory withminimal damage.To accomplish this,we utilize the StarCraftMulti-Agent Challenge(SMAC)environment and train four MARL algorithms:Q-learning with Mixtures of Experts(QMIX),Value-DecompositionNetwork(VDN),Multi-agent Proximal PolicyOptimizer(MAPPO),andMulti-Agent Actor Attention Critic(MAA2C).These algorithms allow multiple agents to cooperate in a specific scenario to achieve the targeted mission.Our results show that the QMIX algorithm outperforms the other three algorithms in the attacking scenario,while the VDN algorithm achieves the best results in the defending scenario.Specifically,the VDNalgorithmreaches the highest value of battle wonmean and the lowest value of dead alliesmean.Our research demonstrates the potential forMARL algorithms to be used in real-world applications,such as controllingmultiple robots to provide helpful services or coordinating teams of agents to accomplish tasks that would be impossible for a human to do.The SMAC environment provides a unique opportunity to test and evaluate MARL algorithms in a challenging and dynamic environment,and our results show that these algorithms can be used to achieve victory with minimal damage.
文摘The disorders caused by the swelling of the soil on the structures have been observed for several years in the city of Rufisque. This article presents the results of the study of swelling kinetics of expansive soils in Rufisque and their prediction based on the hyperbolic rule. The odometer is used as an instrument for measuring swelling and the tests are carried out on some intact samples at their sampling water content. The present study shows that in Rufisque the most swelling layer is marl. The results show two phases of development. The first phase is very fast and represents 77% of the final deformation and the second one is slower. The prediction of the issue by the hyperbolic rule shows that it underestimates the first phase but gives a good prediction of the second phase of the swelling rate. There is a good correlation between the final swelling rates. However, the model gives a bad approximation of the half-swelling time.
文摘Leakage of oil and its derivatives into the soil can change the engineering behavior of soil as well as cause environmental disasters.Also,recovering the contaminated sites into their natural condition and making contaminated materials as both environmentally and geotechnically suitable construction materials need the employment of remediation techniques.Bioremediation,as an efficient,low cost and environmentalfriendly approach,was used in the case of highly plastic clayey soils.To better understand the change in geotechnical properties of highly plastic fine-grained soil due to crude oil contamination and bioremediation,Atterberg limits,compaction,unconfined compression,direct shear,and consolidation tests were conducted on natural,contaminated,and bioremediated soil samples to investigate the effects of contamination and remediation on fine-grained soil properties.Oil contamination reduced maximum dry density(MDD),optimum moisture content(OMC),unconfined compressive strength(UCS),shear strength,swelling pressure,and coefficient of consolidation of soil.In addition,contamination increased the compression and swelling indices and compressibility of soil.Bioremediation reduced soil contamination by about 50%.Moreover,in comparison with contaminated soil,bioremediation reduced the MDD,UCS,swelling index,free swelling and swelling pressure of soil,and also increased OMC,shear strength,cohesion,internal friction angle,failure strain,porosity,compression index,and settlement.Microstructural analyses showed that oil contamination does not alter the soil structure in terms of chemical compounds,elements,and constituent minerals.While it decreased the specific surface area of the soil,and the bioremediation significantly increased the mentioned parameters.Bioremediation resulted in the formation of quasi-fibrous textures and porous and agglomerated structures.As a result,oil contamination affected the mechanical properties of soil negatively,but bioremediation improved these properties.
基金supported by the National Basic Research Program of China(973 Program)(No.2014CB239001).
文摘Due to the complicated lithology in the ES3 Member of the Shahejie Formation in the Shulu sag,Jizhong depression,it is difficult to classify the rock types and characterize the reservoirs at the marl intervals.In this paper,a four-element classification method has been proposed,and seven rock types have been identified by analyzing the mineral composition.The primary rock types are medium-high organic carbonate rocks and medium-high organic shaly-siliceous carbonate rocks.With the methods of field emission scanning electron microscopy,high-pressure mercury intrusion,nitrogen adsorption,and nano-CT,four types of reservoir spaces have been identified,including intra-granular pores,intergranular pores(inter-crystalline pores),organic pores,and micro-fractures.By combining the method of high-pressure mercury intrusion with the method of the nitrogen adsorption,the porosity of the marl has been measured,ranging from 0.73%to 5.39%.The distribution of the pore sizes is bimodal,and the pore types are dominated by micron pores.Through this study,it has been concluded that the sag area to the east of Well ST1H is the favorable area for the development of self-sourced and self-reservoired shale oil.According to the results of geochemical and reservoir analysis,the III Oil Group may have sweet spot layers.
基金supported by the National Key R&D Program of China (2018AAA0101400)the National Natural Science Foundation of China (62173251+3 种基金61921004U1713209)the Natural Science Foundation of Jiangsu Province of China (BK20202006)the Guangdong Provincial Key Laboratory of Intelligent Decision and Cooperative Control。
文摘In this paper, the reinforcement learning method for cooperative multi-agent systems(MAS) with incremental number of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with a specific number of agents, and can learn well-performed policies. However, if there is an increasing number of agents, the previously learned in may not perform well in the current scenario. The new agents need to learn from scratch to find optimal policies with others,which may slow down the learning speed of the whole team. To solve that problem, in this paper, we propose a new algorithm to take full advantage of the historical knowledge which was learned before, and transfer it from the previous agents to the new agents. Since the previous agents have been trained well in the source environment, they are treated as teacher agents in the target environment. Correspondingly, the new agents are called student agents. To enable the student agents to learn from the teacher agents, we first modify the input nodes of the networks for teacher agents to adapt to the current environment. Then, the teacher agents take the observations of the student agents as input, and output the advised actions and values as supervising information. Finally, the student agents combine the reward from the environment and the supervising information from the teacher agents, and learn the optimal policies with modified loss functions. By taking full advantage of the knowledge of teacher agents, the search space for the student agents will be reduced significantly, which can accelerate the learning speed of the holistic system. The proposed algorithm is verified in some multi-agent simulation environments, and its efficiency has been demonstrated by the experiment results.
基金Supported by the PetroChina Science and Technology Project(2021DJ0501,2018A-0105).
文摘In 2022,the risk exploration well Chongtan1(CT1)in the Sichuan Basin revealed commercial oil and gas flow during test in a new zone–the marl of the second submember of the third member of Leikoupo Formation(Lei-32)of Middle Triassic,recording a significant discovery.However,the hydrocarbon accumulation in marl remains unclear,which restricts the selection and deployment of exploration area.Focusing on Well CT1,the hydrocarbon accumulation characteristics of Lei-32 marl are analyzed to clarify the potential zones for exploration.The following findings are obtained.First,according to the geochemical analysis of petroleum and source rocks,oil and gas in the Lei-32 marl of Well CT1 are originated from the same marl.The marl acts as both source rock and reservoir rock.Second,the Lei-32 marl in central Sichuan Basin is of lagoonal facies,with a thickness of 40–130 m,an area of about 40000 km^(2),a hydrocarbon generation intensity of(4–12)×10^(8) m^(3)/km^(2),and an estimated quantity of generated hydrocarbons of 25×10^(12) m^(3).Third,the lagoonal marl reservoirs are widely distributed in central Sichuan Basin.Typically,in Xichong–Yilong,Ziyang–Jianyang and Moxi South,the reservoirs are 20–60 m thick and cover an area of 7500 km^(2).Fourth,hydrocarbons in the lagoonal marl are generated and stored in the Lei-32 marl,which means that marl serves as both source rock and reservoir rock.They represent a new type of unconventional resource,which is worthy of exploring.Fifth,based on the interpretation of 2D and 3D seismic data from central Sichuan Basin,Xichong and Suining are defined as favorable prospects with estimated resources of(2000–3000)×10^(8) m^(3).
基金supported by the Deutsche Forschungsgemeinschaft(DFG,grant PH 189/2-1)
文摘In fractured reservoirs characterized by low matrix permeability,fracture networks control the main fluid flow paths.However,in layered reservoirs,the vertical extension of fractures is often restricted to single layers.In this study,we explored the effect of changing marl/shale thickness on fracture extension using comprehensive field data and numerical modeling.The field data were sampled from coastal exposures of Liassic limestone-marl/shale alternations in Wales and Somerset(Bristol Channel Basin,UK).The vertical fracture traces of more than 4000 fractures were mapped in detail.Six sections were selected to represent a variety of layer thicknesses.Besides the field data also thin sections were analyzed.Numerical models of fracture extension in a two-layer limestone-marl system were based on field data and laboratory measurements of Young's moduli.The modeled principal stress magnitude σ3 along the lithological contact was used as an indication for fracture extension through marls.Field data exhibit good correlation(R^2=0.76) between fracture extension and marl thickness,the thicker the marl layer the fewer fractures propagate through.The model results show that almost no tensile stress reaches the top of the marl layer when the marls are thicker than 30 cm.For marls that are less than 20 cm,the propagation of stress is more dependent on the stiffness of the marls.The higher the contrast between limestone and marl stiffness the lower the stress that is transmitted into the marl layer.In both model experiments and field data the critical marl thickness for fracture extension is ca.15-20 cm.This quantification of critical marl thicknesses can be used to improve predictions of fracture networks and permeability in layered rocks.Up-or downsampling methods often ignore spatially continuous impermeable layers with thicknesses that are under the detection limit of seismic data.However,ignoring these layers can lead to overestimates of the overall permeability.Therefore,the understanding of how fractures propagate and terminate through impermeable layers will help to improve the characterization of conventional reservoirs.
