In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become promine...In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become prominent.VLMs are vulnerable to jailbreak attacks,where attackers craft carefully designed prompts to bypass safety mechanisms,leading them to generate harmful content.To address this,we investigate the alignment between visual inputs and task execution,uncovering locality defects and attention biases in VLMs.Based on these findings,we propose VOTI,a novel jailbreak framework leveraging visual obfuscation and task induction.VOTI subtly embeds malicious keywords within neutral image layouts to evade detection,and breaks down harmful queries into a sequence of subtasks.This approach disperses malicious intent across modalities,exploiting VLMs’over-reliance on local visual cues and their fragility in multi-step reasoning to bypass global safety mechanisms.Implemented as an automated framework,VOTI integrates large language models as red-team assistants to generate and iteratively optimize jailbreak strategies.Extensive experiments across seven mainstream VLMs demonstrate VOTI’s effectiveness,achieving a 73.46%attack success rate on GPT-4o-mini.These results reveal critical vulnerabilities in VLMs,highlighting the urgent need for improving robust defenses and multimodal alignment.展开更多
Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions...Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions.Existing methods can be categorized into motion-level,event-level,and story-level ones based on spatiotemporal granularity.However,single-modal approaches struggle to capture complex behavioral semantics and human factors.Therefore,in recent years,vision-language models(VLMs)have been introduced into this field,providing new research perspectives for VAR.In this paper,we systematically review spatiotemporal hierarchical methods in VAR and explore how the introduction of large models has advanced the field.Additionally,we propose the concept of“Factor”to identify and integrate key information from both visual and textual modalities,enhancing multimodal alignment.We also summarize various multimodal alignment methods and provide in-depth analysis and insights into future research directions.展开更多
human-robot collaboration(HRC)is set to transform the manufacturing paradigm by leveraging the strengths of human flexibility and robot precision.The recent breakthrough of Large Language Models(LLMs)and Vision-Langua...human-robot collaboration(HRC)is set to transform the manufacturing paradigm by leveraging the strengths of human flexibility and robot precision.The recent breakthrough of Large Language Models(LLMs)and Vision-Language Models(VLMs)has motivated the preliminary explorations and adoptions of these models in the smart manufacturing field.However,despite the considerable amount of effort,existing research mainly focused on individual components without a comprehensive perspective to address the full potential of VLMs,especially for HRC in smart manufacturing scenarios.To fill the gap,this work offers a systematic review of the latest advance-ments and applications of VLMs in HRC for smart manu-facturing,which covers the fundamental architectures and pretraining methodologies of LLMs and VLMs,their applications in robotic task planning,navigation,and manipulation,and role in enhancing human-robot skill transfer through multimodal data integration.Lastly,the paper discusses current limitations and future research directions in VLM-based HRC,highlighting the trend in fully realizing the potential of these technologies for smart manufacturing.展开更多
The application of visual-language large models in the field of medical health has gradually become a research focus.The models combine the capability for image understanding and natural language processing,and can si...The application of visual-language large models in the field of medical health has gradually become a research focus.The models combine the capability for image understanding and natural language processing,and can simultaneously process multi-modality data such as medical images and medical reports.These models can not only recognize images,but also understand the semantic relationship between images and texts,effectively realize the integration of medical information,and provide strong support for clinical decision-making and disease diagnosis.The visual-language large model has good performance for specific medical tasks,and also shows strong potential and high intelligence in the general task models.This paper provides a comprehensive review of the visual-language large model in the field of medical health.Specifically,this paper first introduces the basic theoretical basis and technical principles.Then,this paper introduces the specific application scenarios in the field of medical health,including modality fusion,semi-supervised learning,weakly supervised learning,unsupervised learning,cross-domain model and general models.Finally,the challenges including insufficient data,interpretability,and practical deployment are discussed.According to the existing challenges,four potential future development directions are given.展开更多
In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural lang...In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural language processing tasks, such as visual question answering and computer vision applications, including image captioning and image-text retrieval, highlighting their adaptability for complex, multimodal datasets. In this work, we review the landscape of Bootstrapping Language-Image Pre-training (BLIP) and other VLM techniques. A comparative analysis is conducted to assess VLMs’ strengths, limitations, and applicability across tasks while examining challenges such as scalability, data quality, and fine-tuning complexities. The work concludes by outlining potential future directions in VLM research, focusing on enhancing model interpretability, addressing ethical implications, and advancing multimodal integration in real-world applications.展开更多
The advent of large vision-language models(LVLMs)represents a remarkable advance in the quest for artificial general intelligence.However,the models’effectiveness in both specialized and general tasks warrants furthe...The advent of large vision-language models(LVLMs)represents a remarkable advance in the quest for artificial general intelligence.However,the models’effectiveness in both specialized and general tasks warrants further investigation.This paper endeavors to evaluate the competency of popular LVLMs in specialized and general tasks,respectively,aiming to offer a comprehensive understanding of these novel models.To gauge their effectiveness in specialized tasks,we employ six challenging tasks in three different application scenarios:natural,healthcare,and industrial.These six tasks include salient/camouflaged/transparent object detection,as well as polyp detection,skin lesion detection,and industrial anomaly detection.We examine the performance of three recent open-source LVLMs,including MiniGPT-v2,LLaVA-1.5,and Shikra,on both visual recognition and localization in these tasks.Moreover,we conduct empirical investigations utilizing the aforementioned LVLMs together with GPT-4V,assessing their multi-modal understanding capabilities in general tasks including object counting,absurd question answering,affordance reasoning,attribute recognition,and spatial relation reasoning.Our investigations reveal that these LVLMs demonstrate limited proficiency not only in specialized tasks but also in general tasks.We delve deep into this inadequacy and uncover several potential factors,including limited cognition in specialized tasks,object hallucination,text-to-image interference,and decreased robustness in complex problems.We hope that this study can provide useful insights for the future development of LVLMs,helping researchers improve LVLMs for both general and specialized applications.展开更多
Large language models(LLMs),such as ChatGPT,have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains.Recently,large vision-langua...Large language models(LLMs),such as ChatGPT,have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains.Recently,large vision-language models(VLMs)that learn rich vision–language correlation from image–text pairs,like BLIP-2 and GPT-4,have been intensively investigated.However,despite these developments,the application of LLMs and VLMs in image quality assessment(IQA),particularly in medical imaging,remains unexplored.This is valuable for objective performance evaluation and potential supplement or even replacement of radiologists’opinions.To this end,this study intro-duces IQAGPT,an innovative computed tomography(CT)IQA system that integrates image-quality captioning VLM with ChatGPT to generate quality scores and textual reports.First,a CT-IQA dataset comprising 1,000 CT slices with diverse quality levels is professionally annotated and compiled for training and evaluation.To better leverage the capabilities of LLMs,the annotated quality scores are converted into semantically rich text descriptions using a prompt template.Second,the image-quality captioning VLM is fine-tuned on the CT-IQA dataset to generate qual-ity descriptions.The captioning model fuses image and text features through cross-modal attention.Third,based on the quality descriptions,users verbally request ChatGPT to rate image-quality scores or produce radiological qual-ity reports.Results demonstrate the feasibility of assessing image quality using LLMs.The proposed IQAGPT outper-formed GPT-4 and CLIP-IQA,as well as multitask classification and regression models that solely rely on images.展开更多
In the field of satellite imagery, remote sensing image captioning(RSIC) is a hot topic with the challenge of overfitting and difficulty of image and text alignment. To address these issues, this paper proposes a visi...In the field of satellite imagery, remote sensing image captioning(RSIC) is a hot topic with the challenge of overfitting and difficulty of image and text alignment. To address these issues, this paper proposes a vision-language aligning paradigm for RSIC to jointly represent vision and language. First, a new RSIC dataset DIOR-Captions is built for augmenting object detection in optical remote(DIOR) sensing images dataset with manually annotated Chinese and English contents. Second, a Vision-Language aligning model with Cross-modal Attention(VLCA) is presented to generate accurate and abundant bilingual descriptions for remote sensing images. Third, a crossmodal learning network is introduced to address the problem of visual-lingual alignment. Notably, VLCA is also applied to end-toend Chinese captions generation by using the pre-training language model of Chinese. The experiments are carried out with various baselines to validate VLCA on the proposed dataset. The results demonstrate that the proposed algorithm is more descriptive and informative than existing algorithms in producing captions.展开更多
Model evaluation using benchmark datasets is an important method to measure the capability of large language models(LLMs)in specific domains,and it is mainly used to assess the knowledge and reasoning abilities of LLM...Model evaluation using benchmark datasets is an important method to measure the capability of large language models(LLMs)in specific domains,and it is mainly used to assess the knowledge and reasoning abilities of LLMs.Therefore,in order to better assess the capability of LLMs in the agricultural domain,Agri-Eval was proposed as a benchmark for assessing the knowledge and reasoning ability of LLMs in agriculture.The assessment dataset used in Agri-Eval covered seven major disciplines in the agricultural domain:crop science,horticulture,plant protection,animal husbandry,forest science,aquaculture science,and grass science,and contained a total of 2283 questions.Among domestic general-purpose LLMs,DeepSeek R1 performed best with an accuracy rate of 75.49%.In the realm of international general-purpose LLMs,Gemini 2.0 pro exp 0205 standed out as the top performer,achieving an accuracy rate of 74.28%.As an LLMs in agriculture vertical,Shennong V2.0 outperformed all the LLMs in China,and the answer accuracy rate of agricultural knowledge exceeded that of all the existing general-purpose LLMs.The launch of Agri-Eval helped the LLM developers to comprehensively evaluate the model's capability in the field of agriculture through a variety of tasks and tests to promote the development of the LLMs in the field of agriculture.展开更多
We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior ...We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior image segmentation provided by CLIP pre-trained models over ImageNet pre-trained models,but struggles with rough pixel-text score maps for complex scene parsing.We argue that,as they contain all textual information in a dataset,the pixel-text score maps,i.e.,dense prompts,are inevitably mixed with noise.To overcome this challenge,we propose a two-step method.Firstly,we extract visual and language features and perform multi-label classification to identify the most likely categories in the input images.Secondly,based on the top-k categories and confidence scores,our method generates scene tokens which can be treated as adaptive prompts for implicit modeling of scenes,and incorporates them into the visual features fed into the decoder for segmentation.Our method imposes a constraint on prompts and suppresses the probability of irrelevant categories appearing in the scene parsing results.Our method achieves competitive performance,limited by the available visual-language pre-trained models.Our CLIP-SP performs 1.14%better(in terms of mIoU)than DenseCLIP on ADE20K,using a ResNet-50 backbone.展开更多
In this paper,we establish and study a single-species logistic model with impulsive age-selective harvesting.First,we prove the ultimate boundedness of the solutions of the system.Then,we obtain conditions for the asy...In this paper,we establish and study a single-species logistic model with impulsive age-selective harvesting.First,we prove the ultimate boundedness of the solutions of the system.Then,we obtain conditions for the asymptotic stability of the trivial solution and the positive periodic solution.Finally,numerical simulations are presented to validate our results.Our results show that age-selective harvesting is more conducive to sustainable population survival than non-age-selective harvesting.展开更多
In recent years,there has been an increasing need for climate information across diverse sectors of society.This demand has arisen from the necessity to adapt to and mitigate the impacts of climate variability and cha...In recent years,there has been an increasing need for climate information across diverse sectors of society.This demand has arisen from the necessity to adapt to and mitigate the impacts of climate variability and change.Likewise,this period has seen a significant increase in our understanding of the physical processes and mechanisms that drive precipitation and its variability across different regions of Africa.By leveraging a large volume of climate model outputs,numerous studies have investigated the model representation of African precipitation as well as underlying physical processes.These studies have assessed whether the physical processes are well depicted and whether the models are fit for informing mitigation and adaptation strategies.This paper provides a review of the progress in precipitation simulation overAfrica in state-of-the-science climate models and discusses the major issues and challenges that remain.展开更多
The Reynolds-averaged Navier-Stokes(RANS)technique enables critical engineering predictions and is widely adopted.However,since this iterative computation relies on the fixed-point iteration,it may converge to unexpec...The Reynolds-averaged Navier-Stokes(RANS)technique enables critical engineering predictions and is widely adopted.However,since this iterative computation relies on the fixed-point iteration,it may converge to unexpected non-physical phase points in practice.We conduct an analysis on the phase-space characteristics and the fixed-point theory underlying the k-ε turbulence model,and employ the classical Kolmogorov flow as a framework,leveraging its direct numerical simulation(DNS)data to construct a one-dimensional(1D)system under periodic/fixed boundary conditions.The RANS results demonstrate that under periodic boundary conditions,the k-ε model exhibits only a unique trivial fixed point,with asymptotes capturing the phase portraits.The stability of this trivial fixed point is determined by a mathematically derived stability phase diagram,indicating the fact that the k-ε model will never converge to correct values under periodic conditions.In contrast,under fixed boundary conditions,the model can yield a stable non-trivial fixed point.The evolutionary mechanisms and their relationship with boundary condition settings systematically explain the inherent limitations of the k-ε model,i.e.,its deficiency in computing the flow field under periodic boundary conditions and sensitivity to boundary-value specifications under fixed boundary conditions.These conclusions are finally validated with the open-source code OpenFOAM.展开更多
Utilizing finite element analysis,the ballistic protection provided by a combination of perforated D-shaped and base armor plates,collectively referred to as radiator armor,is evaluated.ANSYS Explicit Dynamics is empl...Utilizing finite element analysis,the ballistic protection provided by a combination of perforated D-shaped and base armor plates,collectively referred to as radiator armor,is evaluated.ANSYS Explicit Dynamics is employed to simulate the ballistic impact of 7.62 mm armor-piercing projectiles on Aluminum AA5083-H116 and Steel Secure 500 armors,focusing on the evaluation of material deformation and penetration resistance at varying impact points.While the D-shaped armor plate is penetrated by the armor-piercing projectiles,the combination of the perforated D-shaped and base armor plates successfully halts penetration.A numerical model based on the finite element method is developed using software such as SolidWorks and ANSYS to analyze the interaction between radiator armor and bullet.The perforated design of radiator armor is to maintain airflow for radiator function,with hole sizes smaller than the bullet core diameter to protect radiator assemblies.Predictions are made regarding the brittle fracture resulting from the projectile core′s bending due to asymmetric impact,and the resulting fragments failed to penetrate the perforated base armor plate.Craters are formed on the surface of the perforated D-shaped armor plate due to the impact of projectile fragments.The numerical model accurately predicts hole growth and projectile penetration upon impact with the armor,demonstrating effective protection of the radiator assemblies by the radiator armor.展开更多
Recommendation systems are key to boosting user engagement,satisfaction,and retention,particularly on media platforms where personalized content is vital.Sequential recommendation systems learn from user-item interact...Recommendation systems are key to boosting user engagement,satisfaction,and retention,particularly on media platforms where personalized content is vital.Sequential recommendation systems learn from user-item interactions to predict future items of interest.However,many current methods rely on unique user and item IDs,limiting their ability to represent users and items effectively,especially in zero-shot learning scenarios where training data is scarce.With the rapid development of Large Language Models(LLMs),researchers are exploring their potential to enhance recommendation systems.However,there is a semantic gap between the linguistic semantics of LLMs and the collaborative semantics of recommendation systems,where items are typically indexed by IDs.Moreover,most research focuses on item representations,neglecting personalized user modeling.To address these issues,we propose a sequential recommendation framework using LLMs,called CIT-Rec,a model that integrates Collaborative semantics for user representation and Image and Text information for item representation to enhance Recommendations.Specifically,by aligning intuitive image information with text containing semantic features,we can more accurately represent items,improving item representation quality.We focus not only on item representations but also on user representations.To more precisely capture users’personalized preferences,we use traditional sequential recommendation models to train on users’historical interaction data,effectively capturing behavioral patterns.Finally,by combining LLMs and traditional sequential recommendation models,we allow the LLM to understand linguistic semantics while capturing collaborative semantics.Extensive evaluations on real-world datasets show that our model outperforms baseline methods,effectively combining user interaction history with item visual and textual modalities to provide personalized recommendations.展开更多
The National Geophysical Data Center(NGDC)of the United States has collected aeromagnetic data for input into a series of geomagnetic models to improve model resolution;however,in the Tibetan Plateau region,ground-bas...The National Geophysical Data Center(NGDC)of the United States has collected aeromagnetic data for input into a series of geomagnetic models to improve model resolution;however,in the Tibetan Plateau region,ground-based observations remain insufficient to clearly reflect the characteristics of the region’s lithospheric magnetism.In this study,we evaluate the lithospheric magnetism of the Tibetan Plateau by using a 3D surface spline model based on observations from>200 newly constructed repeat stations(portable stations)to determine the spatial distribution of plateau geomagnetism,as well as its correlation with the tectonic features of the region.We analyze the relationships between M≥5 earthquakes and lithospheric magnetic field variations on the Tibetan Plateau and identify regions susceptible to strong earthquakes.We compare the geomagnetic results with those from an enhanced magnetic model(EMM2015)developed by the NGDC and provide insights into improving lithospheric magnetic field calculations in the Tibetan Plateau region.Further research reveals that these magnetic anomalies exhibit distinct differences from the magnetic-seismic correlation mechanisms observed in other tectonic settings;here,they are governed primarily by the combined effects of compressional magnetism,thermal magnetism,and deep thermal stress.This study provides new evidence of geomagnetic anomalies on the Tibetan Plateau,interprets them physically,and demonstrates their potential for identifying seismic hazard zones on the Plateau.展开更多
(Quasi-)closed-form results for the statistical properties of unmanned aerial vehicle(UAV)airto-ground channels are derived for the first time using a novel spatial-vector-based method from a threedimensional(3-D)arbi...(Quasi-)closed-form results for the statistical properties of unmanned aerial vehicle(UAV)airto-ground channels are derived for the first time using a novel spatial-vector-based method from a threedimensional(3-D)arbitrary-elevation one-cylinder model.The derived results include a closed-form expression for the space-time correlation function and some quasi-closed-form ones for the space-Doppler power spectrum density,the level crossing rate,and the average fading duration,which are shown to be the generalizations of those previously obtained from the two-dimensional(2-D)one-ring model and the 3-D low-elevation one-cylinder model for terrestrial mobile-to-mobile channels.The close agreements between the theoretical results and the simulations as well as the measurements validate the utility of the derived channel statistics.Based on the derived expressions,the impacts of some parameters on the channel characteristics are investigated in an effective,efficient,and explicable way,which leads to a general guideline on the manual parameter estimation from the measurement description.展开更多
Effective management of mining areas in the Luo River Basin,located in the eastern Qinling Mountains,is vital for the integrated protection and restoration needed to support the high-quality development of the Yellow ...Effective management of mining areas in the Luo River Basin,located in the eastern Qinling Mountains,is vital for the integrated protection and restoration needed to support the high-quality development of the Yellow River Basin.Using the‘cupball'model,this study analyzes the limiting factors and restoration characteristics across four mining areas and proposes a conceptual model for selecting appropriate restoration approaches.A second conceptual model is then introduced to address regional development needs,incorporating ecological conservation,safety protection,and people's wellbeing.The applicability of the integrated model selection framework is demonstrated through a case study on the south bank of the Qinglongjian River.The results indicate that:(1)The key limiting factors are similar across cases,but the degree of ecological degradation varies.(2)Mildly degraded areas are represented by a shallower and narrower‘cup',where natural recovery is the preferred approach,whereas moderately and severely degraded systems call for assisted regeneration and ecological reconstruction,respectively.(3)When the restoration models determined based on limiting factors and development needs are consistent,the model is directly applicable;if they differ,the option involving less artificial intervention is preferred;(4)Monitoring of the restored mining area on the Qinglongjian River's south bank confirms significant improvements in soil erosion control and vegetation coverage.This study provides a transferable methodology for balancing resource extraction with ecosystem conservation,offering practical insights for other ecologically vulnerable mining regions.展开更多
Climate model prediction has been improved by enhancing model resolution as well as the implementation of sophisticated physical parameterization and refinement of data assimilation systems[section 6.1 in Wang et al.(...Climate model prediction has been improved by enhancing model resolution as well as the implementation of sophisticated physical parameterization and refinement of data assimilation systems[section 6.1 in Wang et al.(2025)].In relation to seasonal forecasting and climate projection in the East Asian summer monsoon season,proper simulation of the seasonal migration of rain bands by models is a challenging and limiting factor[section 7.1 in Wang et al.(2025)].展开更多
In rock engineering,natural cracks in rock masses subjected to external loads tend to initiate and propagate,leading to potential safety hazards.To investigate the effect of cracking behavior on the mechanical propert...In rock engineering,natural cracks in rock masses subjected to external loads tend to initiate and propagate,leading to potential safety hazards.To investigate the effect of cracking behavior on the mechanical properties of rocks,the cracking processes of pre-cracked rocks have been extensively studied using numerical modeling methods.The peridynamics(PD)exhibits advantages over other numerical methods due to the absence of the requirements for remeshing and external crack growth criterion.However,for modeling pre-cracked rock cracking processes under impact,current PD implementations lack generally applicable rock constitutive models and impact contact models,which leads to difficulties in determining rock material parameters and efficiently calculating impact loads.This paper proposes a non-ordinary state-based peridynamics(NOSBPD)modeling method integrating the Drucker-Prager(DP)plasticity model and an efficient contact model to address the above problems.In the proposed method,the Drucker-Prager plasticity model is integrated into the NOSBPD,thereby equipping NOSBPD with the capability to accurately characterize the nonlinear stress-strain relationship inherent in rocks.An efficient contact model between particles and meshes is designed to calculate the impact loads,which is essentially a coupling method of PD with the finite element method(FEM).The effectiveness of the proposed NOSBPD modeling method is verified by comparison with other numerical methods and experiments.Experimental results indicate that the proposed method can effectively and accurately predict the 3D cracking processes of pre-cracked cracks under impact loading,and the maximum principal stress is the key driver behind wing crack formation in pre-cracked rocks.展开更多
文摘In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become prominent.VLMs are vulnerable to jailbreak attacks,where attackers craft carefully designed prompts to bypass safety mechanisms,leading them to generate harmful content.To address this,we investigate the alignment between visual inputs and task execution,uncovering locality defects and attention biases in VLMs.Based on these findings,we propose VOTI,a novel jailbreak framework leveraging visual obfuscation and task induction.VOTI subtly embeds malicious keywords within neutral image layouts to evade detection,and breaks down harmful queries into a sequence of subtasks.This approach disperses malicious intent across modalities,exploiting VLMs’over-reliance on local visual cues and their fragility in multi-step reasoning to bypass global safety mechanisms.Implemented as an automated framework,VOTI integrates large language models as red-team assistants to generate and iteratively optimize jailbreak strategies.Extensive experiments across seven mainstream VLMs demonstrate VOTI’s effectiveness,achieving a 73.46%attack success rate on GPT-4o-mini.These results reveal critical vulnerabilities in VLMs,highlighting the urgent need for improving robust defenses and multimodal alignment.
基金supported by the Zhejiang Provincial Natural Science Foundation of China(No.LQ23F030001)the National Natural Science Foundation of China(No.62406280)+5 种基金the Autism Research Special Fund of Zhejiang Foundation for Disabled Persons(No.2023008)the Liaoning Province Higher Education Innovative Talents Program Support Project(No.LR2019058)the Liaoning Province Joint Open Fund for Key Scientific and Technological Innovation Bases(No.2021-KF-12-05)the Central Guidance on Local Science and Technology Development Fund of Liaoning Province(No.2023JH6/100100066)the Key Laboratory for Biomedical Engineering of Ministry of Education,Zhejiang University,Chinain part by the Open Research Fund of the State Key Laboratory of Cognitive Neuroscience and Learning.
文摘Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions.Existing methods can be categorized into motion-level,event-level,and story-level ones based on spatiotemporal granularity.However,single-modal approaches struggle to capture complex behavioral semantics and human factors.Therefore,in recent years,vision-language models(VLMs)have been introduced into this field,providing new research perspectives for VAR.In this paper,we systematically review spatiotemporal hierarchical methods in VAR and explore how the introduction of large models has advanced the field.Additionally,we propose the concept of“Factor”to identify and integrate key information from both visual and textual modalities,enhancing multimodal alignment.We also summarize various multimodal alignment methods and provide in-depth analysis and insights into future research directions.
基金Research Institute for Advanced Manufacturing(RIAM)of The Hong Kong Polytechnic University(1-CDJT)Intra-Faculty Interdisciplinary Project 2023/24(1-WZ4N)+6 种基金Research Committee of The Hong Kong Polytechnic UniversityState Key Laboratory of Intelligent Manufacturing Equipment and Technology,Huazhong University of Science and Technology(IMETKF2024010)Guangdong-Hong Kong Technology Cooperation Funding Scheme(GHX/075/22GD)Innovation and Technology Commission(ITC)COMAC International Collaborative Research Project(COMAC-SFGS-2023-3148)General Research Fund from the Research Grants Council of the Hong Kong Special Administrative Region,China(Project Nos.PolyU15210222 and PolyU15206723)Open access funding provided by the Hong Kong Polytechnic University.
文摘human-robot collaboration(HRC)is set to transform the manufacturing paradigm by leveraging the strengths of human flexibility and robot precision.The recent breakthrough of Large Language Models(LLMs)and Vision-Language Models(VLMs)has motivated the preliminary explorations and adoptions of these models in the smart manufacturing field.However,despite the considerable amount of effort,existing research mainly focused on individual components without a comprehensive perspective to address the full potential of VLMs,especially for HRC in smart manufacturing scenarios.To fill the gap,this work offers a systematic review of the latest advance-ments and applications of VLMs in HRC for smart manu-facturing,which covers the fundamental architectures and pretraining methodologies of LLMs and VLMs,their applications in robotic task planning,navigation,and manipulation,and role in enhancing human-robot skill transfer through multimodal data integration.Lastly,the paper discusses current limitations and future research directions in VLM-based HRC,highlighting the trend in fully realizing the potential of these technologies for smart manufacturing.
基金The Natural Science Foundation of Hebei Province(F2024501044).
文摘The application of visual-language large models in the field of medical health has gradually become a research focus.The models combine the capability for image understanding and natural language processing,and can simultaneously process multi-modality data such as medical images and medical reports.These models can not only recognize images,but also understand the semantic relationship between images and texts,effectively realize the integration of medical information,and provide strong support for clinical decision-making and disease diagnosis.The visual-language large model has good performance for specific medical tasks,and also shows strong potential and high intelligence in the general task models.This paper provides a comprehensive review of the visual-language large model in the field of medical health.Specifically,this paper first introduces the basic theoretical basis and technical principles.Then,this paper introduces the specific application scenarios in the field of medical health,including modality fusion,semi-supervised learning,weakly supervised learning,unsupervised learning,cross-domain model and general models.Finally,the challenges including insufficient data,interpretability,and practical deployment are discussed.According to the existing challenges,four potential future development directions are given.
文摘In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural language processing tasks, such as visual question answering and computer vision applications, including image captioning and image-text retrieval, highlighting their adaptability for complex, multimodal datasets. In this work, we review the landscape of Bootstrapping Language-Image Pre-training (BLIP) and other VLM techniques. A comparative analysis is conducted to assess VLMs’ strengths, limitations, and applicability across tasks while examining challenges such as scalability, data quality, and fine-tuning complexities. The work concludes by outlining potential future directions in VLM research, focusing on enhancing model interpretability, addressing ethical implications, and advancing multimodal integration in real-world applications.
基金supported by the National Natural Science Foundation of China(No.62176169)the Fundamental Research Funds for the Central Universities(Nankai University,070-63243150).
文摘The advent of large vision-language models(LVLMs)represents a remarkable advance in the quest for artificial general intelligence.However,the models’effectiveness in both specialized and general tasks warrants further investigation.This paper endeavors to evaluate the competency of popular LVLMs in specialized and general tasks,respectively,aiming to offer a comprehensive understanding of these novel models.To gauge their effectiveness in specialized tasks,we employ six challenging tasks in three different application scenarios:natural,healthcare,and industrial.These six tasks include salient/camouflaged/transparent object detection,as well as polyp detection,skin lesion detection,and industrial anomaly detection.We examine the performance of three recent open-source LVLMs,including MiniGPT-v2,LLaVA-1.5,and Shikra,on both visual recognition and localization in these tasks.Moreover,we conduct empirical investigations utilizing the aforementioned LVLMs together with GPT-4V,assessing their multi-modal understanding capabilities in general tasks including object counting,absurd question answering,affordance reasoning,attribute recognition,and spatial relation reasoning.Our investigations reveal that these LVLMs demonstrate limited proficiency not only in specialized tasks but also in general tasks.We delve deep into this inadequacy and uncover several potential factors,including limited cognition in specialized tasks,object hallucination,text-to-image interference,and decreased robustness in complex problems.We hope that this study can provide useful insights for the future development of LVLMs,helping researchers improve LVLMs for both general and specialized applications.
基金supported in part by the National Natural Science Foundation of China,No.62101136Shanghai Sailing Program,No.21YF1402800National Institutes of Health,Nos.R01CA237267,R01HL151561,R01EB031102,and R01EB032716.
文摘Large language models(LLMs),such as ChatGPT,have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains.Recently,large vision-language models(VLMs)that learn rich vision–language correlation from image–text pairs,like BLIP-2 and GPT-4,have been intensively investigated.However,despite these developments,the application of LLMs and VLMs in image quality assessment(IQA),particularly in medical imaging,remains unexplored.This is valuable for objective performance evaluation and potential supplement or even replacement of radiologists’opinions.To this end,this study intro-duces IQAGPT,an innovative computed tomography(CT)IQA system that integrates image-quality captioning VLM with ChatGPT to generate quality scores and textual reports.First,a CT-IQA dataset comprising 1,000 CT slices with diverse quality levels is professionally annotated and compiled for training and evaluation.To better leverage the capabilities of LLMs,the annotated quality scores are converted into semantically rich text descriptions using a prompt template.Second,the image-quality captioning VLM is fine-tuned on the CT-IQA dataset to generate qual-ity descriptions.The captioning model fuses image and text features through cross-modal attention.Third,based on the quality descriptions,users verbally request ChatGPT to rate image-quality scores or produce radiological qual-ity reports.Results demonstrate the feasibility of assessing image quality using LLMs.The proposed IQAGPT outper-formed GPT-4 and CLIP-IQA,as well as multitask classification and regression models that solely rely on images.
基金supported by the National Natural Science Foundation of China (61702528,61806212)。
文摘In the field of satellite imagery, remote sensing image captioning(RSIC) is a hot topic with the challenge of overfitting and difficulty of image and text alignment. To address these issues, this paper proposes a vision-language aligning paradigm for RSIC to jointly represent vision and language. First, a new RSIC dataset DIOR-Captions is built for augmenting object detection in optical remote(DIOR) sensing images dataset with manually annotated Chinese and English contents. Second, a Vision-Language aligning model with Cross-modal Attention(VLCA) is presented to generate accurate and abundant bilingual descriptions for remote sensing images. Third, a crossmodal learning network is introduced to address the problem of visual-lingual alignment. Notably, VLCA is also applied to end-toend Chinese captions generation by using the pre-training language model of Chinese. The experiments are carried out with various baselines to validate VLCA on the proposed dataset. The results demonstrate that the proposed algorithm is more descriptive and informative than existing algorithms in producing captions.
文摘Model evaluation using benchmark datasets is an important method to measure the capability of large language models(LLMs)in specific domains,and it is mainly used to assess the knowledge and reasoning abilities of LLMs.Therefore,in order to better assess the capability of LLMs in the agricultural domain,Agri-Eval was proposed as a benchmark for assessing the knowledge and reasoning ability of LLMs in agriculture.The assessment dataset used in Agri-Eval covered seven major disciplines in the agricultural domain:crop science,horticulture,plant protection,animal husbandry,forest science,aquaculture science,and grass science,and contained a total of 2283 questions.Among domestic general-purpose LLMs,DeepSeek R1 performed best with an accuracy rate of 75.49%.In the realm of international general-purpose LLMs,Gemini 2.0 pro exp 0205 standed out as the top performer,achieving an accuracy rate of 74.28%.As an LLMs in agriculture vertical,Shennong V2.0 outperformed all the LLMs in China,and the answer accuracy rate of agricultural knowledge exceeded that of all the existing general-purpose LLMs.The launch of Agri-Eval helped the LLM developers to comprehensively evaluate the model's capability in the field of agriculture through a variety of tasks and tests to promote the development of the LLMs in the field of agriculture.
文摘We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior image segmentation provided by CLIP pre-trained models over ImageNet pre-trained models,but struggles with rough pixel-text score maps for complex scene parsing.We argue that,as they contain all textual information in a dataset,the pixel-text score maps,i.e.,dense prompts,are inevitably mixed with noise.To overcome this challenge,we propose a two-step method.Firstly,we extract visual and language features and perform multi-label classification to identify the most likely categories in the input images.Secondly,based on the top-k categories and confidence scores,our method generates scene tokens which can be treated as adaptive prompts for implicit modeling of scenes,and incorporates them into the visual features fed into the decoder for segmentation.Our method imposes a constraint on prompts and suppresses the probability of irrelevant categories appearing in the scene parsing results.Our method achieves competitive performance,limited by the available visual-language pre-trained models.Our CLIP-SP performs 1.14%better(in terms of mIoU)than DenseCLIP on ADE20K,using a ResNet-50 backbone.
基金Supported by the National Natural Science Foundation of China(12261018)Universities Key Laboratory of Mathematical Modeling and Data Mining in Guizhou Province(2023013)。
文摘In this paper,we establish and study a single-species logistic model with impulsive age-selective harvesting.First,we prove the ultimate boundedness of the solutions of the system.Then,we obtain conditions for the asymptotic stability of the trivial solution and the positive periodic solution.Finally,numerical simulations are presented to validate our results.Our results show that age-selective harvesting is more conducive to sustainable population survival than non-age-selective harvesting.
基金the World Climate Research Programme(WCRP),Climate Variability and Predictability(CLIVAR),and Global Energy and Water Exchanges(GEWEX)for facilitating the coordination of African monsoon researchsupport from the Center for Earth System Modeling,Analysis,and Data at the Pennsylvania State Universitythe support of the Office of Science of the U.S.Department of Energy Biological and Environmental Research as part of the Regional&Global Model Analysis(RGMA)program area。
文摘In recent years,there has been an increasing need for climate information across diverse sectors of society.This demand has arisen from the necessity to adapt to and mitigate the impacts of climate variability and change.Likewise,this period has seen a significant increase in our understanding of the physical processes and mechanisms that drive precipitation and its variability across different regions of Africa.By leveraging a large volume of climate model outputs,numerous studies have investigated the model representation of African precipitation as well as underlying physical processes.These studies have assessed whether the physical processes are well depicted and whether the models are fit for informing mitigation and adaptation strategies.This paper provides a review of the progress in precipitation simulation overAfrica in state-of-the-science climate models and discusses the major issues and challenges that remain.
基金Project supported by the National Natural Science Foundation of China(Nos.12372214 and U2341231)。
文摘The Reynolds-averaged Navier-Stokes(RANS)technique enables critical engineering predictions and is widely adopted.However,since this iterative computation relies on the fixed-point iteration,it may converge to unexpected non-physical phase points in practice.We conduct an analysis on the phase-space characteristics and the fixed-point theory underlying the k-ε turbulence model,and employ the classical Kolmogorov flow as a framework,leveraging its direct numerical simulation(DNS)data to construct a one-dimensional(1D)system under periodic/fixed boundary conditions.The RANS results demonstrate that under periodic boundary conditions,the k-ε model exhibits only a unique trivial fixed point,with asymptotes capturing the phase portraits.The stability of this trivial fixed point is determined by a mathematically derived stability phase diagram,indicating the fact that the k-ε model will never converge to correct values under periodic conditions.In contrast,under fixed boundary conditions,the model can yield a stable non-trivial fixed point.The evolutionary mechanisms and their relationship with boundary condition settings systematically explain the inherent limitations of the k-ε model,i.e.,its deficiency in computing the flow field under periodic boundary conditions and sensitivity to boundary-value specifications under fixed boundary conditions.These conclusions are finally validated with the open-source code OpenFOAM.
文摘Utilizing finite element analysis,the ballistic protection provided by a combination of perforated D-shaped and base armor plates,collectively referred to as radiator armor,is evaluated.ANSYS Explicit Dynamics is employed to simulate the ballistic impact of 7.62 mm armor-piercing projectiles on Aluminum AA5083-H116 and Steel Secure 500 armors,focusing on the evaluation of material deformation and penetration resistance at varying impact points.While the D-shaped armor plate is penetrated by the armor-piercing projectiles,the combination of the perforated D-shaped and base armor plates successfully halts penetration.A numerical model based on the finite element method is developed using software such as SolidWorks and ANSYS to analyze the interaction between radiator armor and bullet.The perforated design of radiator armor is to maintain airflow for radiator function,with hole sizes smaller than the bullet core diameter to protect radiator assemblies.Predictions are made regarding the brittle fracture resulting from the projectile core′s bending due to asymmetric impact,and the resulting fragments failed to penetrate the perforated base armor plate.Craters are formed on the surface of the perforated D-shaped armor plate due to the impact of projectile fragments.The numerical model accurately predicts hole growth and projectile penetration upon impact with the armor,demonstrating effective protection of the radiator assemblies by the radiator armor.
基金supported by the National Key R&D Program of China[2022YFF0902703]the State Administration for Market Regulation Science and Technology Plan Project(2024MK033).
文摘Recommendation systems are key to boosting user engagement,satisfaction,and retention,particularly on media platforms where personalized content is vital.Sequential recommendation systems learn from user-item interactions to predict future items of interest.However,many current methods rely on unique user and item IDs,limiting their ability to represent users and items effectively,especially in zero-shot learning scenarios where training data is scarce.With the rapid development of Large Language Models(LLMs),researchers are exploring their potential to enhance recommendation systems.However,there is a semantic gap between the linguistic semantics of LLMs and the collaborative semantics of recommendation systems,where items are typically indexed by IDs.Moreover,most research focuses on item representations,neglecting personalized user modeling.To address these issues,we propose a sequential recommendation framework using LLMs,called CIT-Rec,a model that integrates Collaborative semantics for user representation and Image and Text information for item representation to enhance Recommendations.Specifically,by aligning intuitive image information with text containing semantic features,we can more accurately represent items,improving item representation quality.We focus not only on item representations but also on user representations.To more precisely capture users’personalized preferences,we use traditional sequential recommendation models to train on users’historical interaction data,effectively capturing behavioral patterns.Finally,by combining LLMs and traditional sequential recommendation models,we allow the LLM to understand linguistic semantics while capturing collaborative semantics.Extensive evaluations on real-world datasets show that our model outperforms baseline methods,effectively combining user interaction history with item visual and textual modalities to provide personalized recommendations.
基金supported by the CAS Pioneer Hundred Talents Program and Second Tibetan Plateau Scientific Expedition Research Program(2019QZKK0708)as well as the Basic Research Program of Qinghai Province:Lithospheric Geomagnetic Field of the Qinghai-Tibet Plateau and the Relationship with Strong Earthquakes(2021-ZJ-969Q).
文摘The National Geophysical Data Center(NGDC)of the United States has collected aeromagnetic data for input into a series of geomagnetic models to improve model resolution;however,in the Tibetan Plateau region,ground-based observations remain insufficient to clearly reflect the characteristics of the region’s lithospheric magnetism.In this study,we evaluate the lithospheric magnetism of the Tibetan Plateau by using a 3D surface spline model based on observations from>200 newly constructed repeat stations(portable stations)to determine the spatial distribution of plateau geomagnetism,as well as its correlation with the tectonic features of the region.We analyze the relationships between M≥5 earthquakes and lithospheric magnetic field variations on the Tibetan Plateau and identify regions susceptible to strong earthquakes.We compare the geomagnetic results with those from an enhanced magnetic model(EMM2015)developed by the NGDC and provide insights into improving lithospheric magnetic field calculations in the Tibetan Plateau region.Further research reveals that these magnetic anomalies exhibit distinct differences from the magnetic-seismic correlation mechanisms observed in other tectonic settings;here,they are governed primarily by the combined effects of compressional magnetism,thermal magnetism,and deep thermal stress.This study provides new evidence of geomagnetic anomalies on the Tibetan Plateau,interprets them physically,and demonstrates their potential for identifying seismic hazard zones on the Plateau.
基金supported in part by the National Key Research and Development Program of China(2021YFB2900501)in part by the Shaanxi Science and Technology Innovation Team(2023-CX-TD-03)+3 种基金in part by the Science and Technology Program of Shaanxi Province(2021GXLH-Z-038)in part by the Natural Science Foundation of Hunan Province(2023JJ40607 and 2023JJ50045)in part by the Scientific Research Foundation of Hunan Provincial Education Department(23B0713 and 24B0603)in part by the National Natural Science Foundation of China(62401371,62101275,and 62372070).
文摘(Quasi-)closed-form results for the statistical properties of unmanned aerial vehicle(UAV)airto-ground channels are derived for the first time using a novel spatial-vector-based method from a threedimensional(3-D)arbitrary-elevation one-cylinder model.The derived results include a closed-form expression for the space-time correlation function and some quasi-closed-form ones for the space-Doppler power spectrum density,the level crossing rate,and the average fading duration,which are shown to be the generalizations of those previously obtained from the two-dimensional(2-D)one-ring model and the 3-D low-elevation one-cylinder model for terrestrial mobile-to-mobile channels.The close agreements between the theoretical results and the simulations as well as the measurements validate the utility of the derived channel statistics.Based on the derived expressions,the impacts of some parameters on the channel characteristics are investigated in an effective,efficient,and explicable way,which leads to a general guideline on the manual parameter estimation from the measurement description.
基金supported by Special major projects for research and development of Henan Provincial(Science and Technology Research Project)(No.252102321104)Humanities and Social Sciences Youth Foundation,Ministry of Education(24YJCZH410)。
文摘Effective management of mining areas in the Luo River Basin,located in the eastern Qinling Mountains,is vital for the integrated protection and restoration needed to support the high-quality development of the Yellow River Basin.Using the‘cupball'model,this study analyzes the limiting factors and restoration characteristics across four mining areas and proposes a conceptual model for selecting appropriate restoration approaches.A second conceptual model is then introduced to address regional development needs,incorporating ecological conservation,safety protection,and people's wellbeing.The applicability of the integrated model selection framework is demonstrated through a case study on the south bank of the Qinglongjian River.The results indicate that:(1)The key limiting factors are similar across cases,but the degree of ecological degradation varies.(2)Mildly degraded areas are represented by a shallower and narrower‘cup',where natural recovery is the preferred approach,whereas moderately and severely degraded systems call for assisted regeneration and ecological reconstruction,respectively.(3)When the restoration models determined based on limiting factors and development needs are consistent,the model is directly applicable;if they differ,the option involving less artificial intervention is preferred;(4)Monitoring of the restored mining area on the Qinglongjian River's south bank confirms significant improvements in soil erosion control and vegetation coverage.This study provides a transferable methodology for balancing resource extraction with ecosystem conservation,offering practical insights for other ecologically vulnerable mining regions.
文摘Climate model prediction has been improved by enhancing model resolution as well as the implementation of sophisticated physical parameterization and refinement of data assimilation systems[section 6.1 in Wang et al.(2025)].In relation to seasonal forecasting and climate projection in the East Asian summer monsoon season,proper simulation of the seasonal migration of rain bands by models is a challenging and limiting factor[section 7.1 in Wang et al.(2025)].
基金support from the National Natural Science Foundation of China(Grant Nos.42277161 and 42230709).
文摘In rock engineering,natural cracks in rock masses subjected to external loads tend to initiate and propagate,leading to potential safety hazards.To investigate the effect of cracking behavior on the mechanical properties of rocks,the cracking processes of pre-cracked rocks have been extensively studied using numerical modeling methods.The peridynamics(PD)exhibits advantages over other numerical methods due to the absence of the requirements for remeshing and external crack growth criterion.However,for modeling pre-cracked rock cracking processes under impact,current PD implementations lack generally applicable rock constitutive models and impact contact models,which leads to difficulties in determining rock material parameters and efficiently calculating impact loads.This paper proposes a non-ordinary state-based peridynamics(NOSBPD)modeling method integrating the Drucker-Prager(DP)plasticity model and an efficient contact model to address the above problems.In the proposed method,the Drucker-Prager plasticity model is integrated into the NOSBPD,thereby equipping NOSBPD with the capability to accurately characterize the nonlinear stress-strain relationship inherent in rocks.An efficient contact model between particles and meshes is designed to calculate the impact loads,which is essentially a coupling method of PD with the finite element method(FEM).The effectiveness of the proposed NOSBPD modeling method is verified by comparison with other numerical methods and experiments.Experimental results indicate that the proposed method can effectively and accurately predict the 3D cracking processes of pre-cracked cracks under impact loading,and the maximum principal stress is the key driver behind wing crack formation in pre-cracked rocks.
