Existing methods for tracing water pollution sources typically integrate three-dimensional excitationemission matrix(3D-EEM)fluorescence spectroscopy with similarity-based matching algorithms.However,these approaches ...Existing methods for tracing water pollution sources typically integrate three-dimensional excitationemission matrix(3D-EEM)fluorescence spectroscopy with similarity-based matching algorithms.However,these approaches exhibit high error rates in borderline cases and necessitate expert manual review,which limits scalability and introduces inconsistencies between algorithmic outputs and expert judgment.To address these limitations,we propose a large vision-language model(VLM)designed as an“expert agent”to automatically refine similarity scores,ensuring alignment with expert decisions and overcoming key application bottlenecks.The model consists of two core components:(1)rule-based similarity calculation module generate initial spectral similarity scores,and(2)pre-trained large vision-language model fine-tuned via supervised learning and reinforcement learning with human feedback(RLHF)to emulate expert assessments.To facilitate training and evaluation,we introduce two expert-annotated datasets,Spec1k and SpecReason,which capture both quantitative corrections and qualitative reasoning patterns,allowing the model to emulate expert decision-making processes.Experimental results demonstrate that our method achieves 81.45%source attribution accuracy,38.24%higher than rule-based and machine learning baselines.Real-world deployment further validates its effectiveness.展开更多
In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become promine...In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become prominent.VLMs are vulnerable to jailbreak attacks,where attackers craft carefully designed prompts to bypass safety mechanisms,leading them to generate harmful content.To address this,we investigate the alignment between visual inputs and task execution,uncovering locality defects and attention biases in VLMs.Based on these findings,we propose VOTI,a novel jailbreak framework leveraging visual obfuscation and task induction.VOTI subtly embeds malicious keywords within neutral image layouts to evade detection,and breaks down harmful queries into a sequence of subtasks.This approach disperses malicious intent across modalities,exploiting VLMs’over-reliance on local visual cues and their fragility in multi-step reasoning to bypass global safety mechanisms.Implemented as an automated framework,VOTI integrates large language models as red-team assistants to generate and iteratively optimize jailbreak strategies.Extensive experiments across seven mainstream VLMs demonstrate VOTI’s effectiveness,achieving a 73.46%attack success rate on GPT-4o-mini.These results reveal critical vulnerabilities in VLMs,highlighting the urgent need for improving robust defenses and multimodal alignment.展开更多
Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions...Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions.Existing methods can be categorized into motion-level,event-level,and story-level ones based on spatiotemporal granularity.However,single-modal approaches struggle to capture complex behavioral semantics and human factors.Therefore,in recent years,vision-language models(VLMs)have been introduced into this field,providing new research perspectives for VAR.In this paper,we systematically review spatiotemporal hierarchical methods in VAR and explore how the introduction of large models has advanced the field.Additionally,we propose the concept of“Factor”to identify and integrate key information from both visual and textual modalities,enhancing multimodal alignment.We also summarize various multimodal alignment methods and provide in-depth analysis and insights into future research directions.展开更多
This paper investigates the potential of Vision-Language Models(VLMs)to enhance Human–Vehicle Interaction(HVI)in Autonomous Driving(AD)scenarios,particularly in interactions between vehicles and other traffic partici...This paper investigates the potential of Vision-Language Models(VLMs)to enhance Human–Vehicle Interaction(HVI)in Autonomous Driving(AD)scenarios,particularly in interactions between vehicles and other traffic participants,with a focus on rationality and safety in external HVI.Leveraging recent advancements in large language models,VLMs demonstrate remarkable capabilities in understanding real-world contexts and generating significant interest in HVI applications.This paper provides an overview of AD,HVI,and VLMs,along with the historical context of large language model applications in HVI.The HVI discussed herein involves dynamic game processes encompassing perception and decision-making between vehicles and traffic participants,such as pedestrians.Furthermore,we examine the perceptual challenges associated with applying VLMs to HVI and compile relevant datasets.This research fills a gap in the existing literature by systematically analyzing the current status,challenges,and future opportunities of VLM applications in HVI.To advance VLM integration in AD,various implementation strategies are discussed.The findings highlight the potential of VLMs to transform HVI in AD,improving both passenger experience and driving safety.Overall,this study contributes to a comprehensive understanding of VLM applications in HVI and provides insights to guide future research and development.展开更多
human-robot collaboration(HRC)is set to transform the manufacturing paradigm by leveraging the strengths of human flexibility and robot precision.The recent breakthrough of Large Language Models(LLMs)and Vision-Langua...human-robot collaboration(HRC)is set to transform the manufacturing paradigm by leveraging the strengths of human flexibility and robot precision.The recent breakthrough of Large Language Models(LLMs)and Vision-Language Models(VLMs)has motivated the preliminary explorations and adoptions of these models in the smart manufacturing field.However,despite the considerable amount of effort,existing research mainly focused on individual components without a comprehensive perspective to address the full potential of VLMs,especially for HRC in smart manufacturing scenarios.To fill the gap,this work offers a systematic review of the latest advance-ments and applications of VLMs in HRC for smart manu-facturing,which covers the fundamental architectures and pretraining methodologies of LLMs and VLMs,their applications in robotic task planning,navigation,and manipulation,and role in enhancing human-robot skill transfer through multimodal data integration.Lastly,the paper discusses current limitations and future research directions in VLM-based HRC,highlighting the trend in fully realizing the potential of these technologies for smart manufacturing.展开更多
In recent years,Vision-Language Models(VLMs)have emerged as a significant breakthrough in multimodal learning,demonstrating remarkable progress in tasks such as image-text alignment,image generation,and semantic reaso...In recent years,Vision-Language Models(VLMs)have emerged as a significant breakthrough in multimodal learning,demonstrating remarkable progress in tasks such as image-text alignment,image generation,and semantic reasoning.This paper systematically reviews current VLM pretraining methodologies,including contrastive learning and generative paradigms,while providing an in-depth analysis of efficient transfer learning strategies such as prompt tuning,LoRA,and adapter modules.Through representative models like CLIP,BLIP,and GIT,we examine their practical applications in visual grounding,imagetext retrieval,visual question answering,affective computing,and embodied AI.Furthermore,we identify persistent challenges in fine-grained semantic modeling,cross-modal reasoning,and cross-lingual transfer.Finally,we envision future trends in unified architectures,multimodal reinforcement learning,and domain adaptation,aiming to provide systematic reference and technical insights for subsequent research.展开更多
The application of visual-language large models in the field of medical health has gradually become a research focus.The models combine the capability for image understanding and natural language processing,and can si...The application of visual-language large models in the field of medical health has gradually become a research focus.The models combine the capability for image understanding and natural language processing,and can simultaneously process multi-modality data such as medical images and medical reports.These models can not only recognize images,but also understand the semantic relationship between images and texts,effectively realize the integration of medical information,and provide strong support for clinical decision-making and disease diagnosis.The visual-language large model has good performance for specific medical tasks,and also shows strong potential and high intelligence in the general task models.This paper provides a comprehensive review of the visual-language large model in the field of medical health.Specifically,this paper first introduces the basic theoretical basis and technical principles.Then,this paper introduces the specific application scenarios in the field of medical health,including modality fusion,semi-supervised learning,weakly supervised learning,unsupervised learning,cross-domain model and general models.Finally,the challenges including insufficient data,interpretability,and practical deployment are discussed.According to the existing challenges,four potential future development directions are given.展开更多
In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural lang...In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural language processing tasks, such as visual question answering and computer vision applications, including image captioning and image-text retrieval, highlighting their adaptability for complex, multimodal datasets. In this work, we review the landscape of Bootstrapping Language-Image Pre-training (BLIP) and other VLM techniques. A comparative analysis is conducted to assess VLMs’ strengths, limitations, and applicability across tasks while examining challenges such as scalability, data quality, and fine-tuning complexities. The work concludes by outlining potential future directions in VLM research, focusing on enhancing model interpretability, addressing ethical implications, and advancing multimodal integration in real-world applications.展开更多
The advent of large vision-language models(LVLMs)represents a remarkable advance in the quest for artificial general intelligence.However,the models’effectiveness in both specialized and general tasks warrants furthe...The advent of large vision-language models(LVLMs)represents a remarkable advance in the quest for artificial general intelligence.However,the models’effectiveness in both specialized and general tasks warrants further investigation.This paper endeavors to evaluate the competency of popular LVLMs in specialized and general tasks,respectively,aiming to offer a comprehensive understanding of these novel models.To gauge their effectiveness in specialized tasks,we employ six challenging tasks in three different application scenarios:natural,healthcare,and industrial.These six tasks include salient/camouflaged/transparent object detection,as well as polyp detection,skin lesion detection,and industrial anomaly detection.We examine the performance of three recent open-source LVLMs,including MiniGPT-v2,LLaVA-1.5,and Shikra,on both visual recognition and localization in these tasks.Moreover,we conduct empirical investigations utilizing the aforementioned LVLMs together with GPT-4V,assessing their multi-modal understanding capabilities in general tasks including object counting,absurd question answering,affordance reasoning,attribute recognition,and spatial relation reasoning.Our investigations reveal that these LVLMs demonstrate limited proficiency not only in specialized tasks but also in general tasks.We delve deep into this inadequacy and uncover several potential factors,including limited cognition in specialized tasks,object hallucination,text-to-image interference,and decreased robustness in complex problems.We hope that this study can provide useful insights for the future development of LVLMs,helping researchers improve LVLMs for both general and specialized applications.展开更多
Large language models(LLMs),such as ChatGPT,have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains.Recently,large vision-langua...Large language models(LLMs),such as ChatGPT,have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains.Recently,large vision-language models(VLMs)that learn rich vision–language correlation from image–text pairs,like BLIP-2 and GPT-4,have been intensively investigated.However,despite these developments,the application of LLMs and VLMs in image quality assessment(IQA),particularly in medical imaging,remains unexplored.This is valuable for objective performance evaluation and potential supplement or even replacement of radiologists’opinions.To this end,this study intro-duces IQAGPT,an innovative computed tomography(CT)IQA system that integrates image-quality captioning VLM with ChatGPT to generate quality scores and textual reports.First,a CT-IQA dataset comprising 1,000 CT slices with diverse quality levels is professionally annotated and compiled for training and evaluation.To better leverage the capabilities of LLMs,the annotated quality scores are converted into semantically rich text descriptions using a prompt template.Second,the image-quality captioning VLM is fine-tuned on the CT-IQA dataset to generate qual-ity descriptions.The captioning model fuses image and text features through cross-modal attention.Third,based on the quality descriptions,users verbally request ChatGPT to rate image-quality scores or produce radiological qual-ity reports.Results demonstrate the feasibility of assessing image quality using LLMs.The proposed IQAGPT outper-formed GPT-4 and CLIP-IQA,as well as multitask classification and regression models that solely rely on images.展开更多
In the field of satellite imagery, remote sensing image captioning(RSIC) is a hot topic with the challenge of overfitting and difficulty of image and text alignment. To address these issues, this paper proposes a visi...In the field of satellite imagery, remote sensing image captioning(RSIC) is a hot topic with the challenge of overfitting and difficulty of image and text alignment. To address these issues, this paper proposes a vision-language aligning paradigm for RSIC to jointly represent vision and language. First, a new RSIC dataset DIOR-Captions is built for augmenting object detection in optical remote(DIOR) sensing images dataset with manually annotated Chinese and English contents. Second, a Vision-Language aligning model with Cross-modal Attention(VLCA) is presented to generate accurate and abundant bilingual descriptions for remote sensing images. Third, a crossmodal learning network is introduced to address the problem of visual-lingual alignment. Notably, VLCA is also applied to end-toend Chinese captions generation by using the pre-training language model of Chinese. The experiments are carried out with various baselines to validate VLCA on the proposed dataset. The results demonstrate that the proposed algorithm is more descriptive and informative than existing algorithms in producing captions.展开更多
Rural domestic sewage treatment is critical for environmental protection.This study defines the spatial pattern of villages from the perspective of rural sewage treatment and develops an integrated decision-making sys...Rural domestic sewage treatment is critical for environmental protection.This study defines the spatial pattern of villages from the perspective of rural sewage treatment and develops an integrated decision-making system to propose a sewage treatment mode and scheme suitable for local conditions.By considering the village spatial layout and terrain factors,a decision tree model of residential density and terrain type was constructed with accuracies of 76.47%and 96.00%,respectively.Combined with binary classification probability unit regression,an appropriate sewage treatment mode for the village was determined with 87.00%accuracy.The Analytic Hierarchy Process(AHP),combined with the Technique for Order Preference(TOPSIS)by Similarity to an Ideal Solution model,formed the basis for optimal treatment process selection under different emission standards.Verification was conducted in 542 villages across three counties of the Inner Mongolia Autonomous Region,focusing on the standard effluent effect(0.3773),low investment cost(0.3196),and high standard effluent effect(0.5115)to determine the best treatment process for the same emission standard under different needs.The annual environmental and carbon emission benefits of sewage treatment in these villages were estimated.This model matches village density,geographic feature,and social development level,and provides scientific support and a theoretical basis for rural sewage treatment decision-making.展开更多
We present a comprehensive description and benchmark evaluation of the global–regional chemical transport model called the Emission and Atmospheric Processes Integrated and Coupled Community(EPICC)model.The framework...We present a comprehensive description and benchmark evaluation of the global–regional chemical transport model called the Emission and Atmospheric Processes Integrated and Coupled Community(EPICC)model.The framework incorporates(1)grid configuration,(2)transport dynamics,(3)chemical mechanisms,(4)aerosol processes,(5)wet/dry deposition parameterizations,and(6)heterogeneous chemistry treatments associated with sulfate,nitrous acid(HONO)chemistry,and aerosol/cloud–photolysis interactions(APIs/CPIs).Openly shared with the atmospheric research community,the model facilitates integration of advanced physicochemical schemes to enhance simulation accuracy.Globally,the model demonstrates realistic representations of ozone(O_(3))and aerosol optical depth.The EPICC model generally demonstrates robust performance in simulating regional concentrations of O_(3) and PM_(2.5)(and its components)in China.It successfully captures vertical profiles of both global and regional O_(3).Notably,the model mitigates frequently reported sulfate underestimations in highly industrialized regions of China.The model accurately captures two regional severe pollution episodes observed in eastern China(January/June 2021).Sensitivity experiments highlight the critical roles of heterogeneous chemical mechanisms associated with sulfate,HONO chemistry,APIs,and CPIs in capturing PM_(2.5) and O_(3) concentrations in China.Improved sulfate mechanisms result in an increase of approximately 32.4%(2.8μg m^(−3))in simulated winter sulfate concentrations when observations exceed 10μg m^(−3).Enhanced HONO elevates winter O_(3) and PM_(2.5) by≤20 and≤10μg m^(−3),respectively.Overall,CPIs dominate over APIs in improving O_(3) and PM_(2.5) simulations across China.Locally,APIs mitigate PM_(2.5) and O_(3) discrepancies in the Sichuan Basin.Seasonal cloud–chemistry coupling explains the weaker impact of PM_(2.5) in summer.展开更多
Model evaluation using benchmark datasets is an important method to measure the capability of large language models(LLMs)in specific domains,and it is mainly used to assess the knowledge and reasoning abilities of LLM...Model evaluation using benchmark datasets is an important method to measure the capability of large language models(LLMs)in specific domains,and it is mainly used to assess the knowledge and reasoning abilities of LLMs.Therefore,in order to better assess the capability of LLMs in the agricultural domain,Agri-Eval was proposed as a benchmark for assessing the knowledge and reasoning ability of LLMs in agriculture.The assessment dataset used in Agri-Eval covered seven major disciplines in the agricultural domain:crop science,horticulture,plant protection,animal husbandry,forest science,aquaculture science,and grass science,and contained a total of 2283 questions.Among domestic general-purpose LLMs,DeepSeek R1 performed best with an accuracy rate of 75.49%.In the realm of international general-purpose LLMs,Gemini 2.0 pro exp 0205 standed out as the top performer,achieving an accuracy rate of 74.28%.As an LLMs in agriculture vertical,Shennong V2.0 outperformed all the LLMs in China,and the answer accuracy rate of agricultural knowledge exceeded that of all the existing general-purpose LLMs.The launch of Agri-Eval helped the LLM developers to comprehensively evaluate the model's capability in the field of agriculture through a variety of tasks and tests to promote the development of the LLMs in the field of agriculture.展开更多
We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior ...We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior image segmentation provided by CLIP pre-trained models over ImageNet pre-trained models,but struggles with rough pixel-text score maps for complex scene parsing.We argue that,as they contain all textual information in a dataset,the pixel-text score maps,i.e.,dense prompts,are inevitably mixed with noise.To overcome this challenge,we propose a two-step method.Firstly,we extract visual and language features and perform multi-label classification to identify the most likely categories in the input images.Secondly,based on the top-k categories and confidence scores,our method generates scene tokens which can be treated as adaptive prompts for implicit modeling of scenes,and incorporates them into the visual features fed into the decoder for segmentation.Our method imposes a constraint on prompts and suppresses the probability of irrelevant categories appearing in the scene parsing results.Our method achieves competitive performance,limited by the available visual-language pre-trained models.Our CLIP-SP performs 1.14%better(in terms of mIoU)than DenseCLIP on ADE20K,using a ResNet-50 backbone.展开更多
In this paper,we establish and study a single-species logistic model with impulsive age-selective harvesting.First,we prove the ultimate boundedness of the solutions of the system.Then,we obtain conditions for the asy...In this paper,we establish and study a single-species logistic model with impulsive age-selective harvesting.First,we prove the ultimate boundedness of the solutions of the system.Then,we obtain conditions for the asymptotic stability of the trivial solution and the positive periodic solution.Finally,numerical simulations are presented to validate our results.Our results show that age-selective harvesting is more conducive to sustainable population survival than non-age-selective harvesting.展开更多
The proliferation of high-dimensional data and the widespread use of complex models present central challenges in contemporary statistics and data science.Dimension reduction and model checking,as two foundational pil...The proliferation of high-dimensional data and the widespread use of complex models present central challenges in contemporary statistics and data science.Dimension reduction and model checking,as two foundational pillars supporting scientific inference and data-driven decisionmaking,have evolved through the collective wisdom of generations of statisticians.This special issue,titled"Recent Developments in Dimension Reduction and Model Checking for regressions",not only aims to showcase cutting-edge advances in the field but also carries a distinct sense of academic homage to honor the groundbreaking and enduring contributions of Professor Lixing Zhu,a leading scholar whose work has profoundly shaped both areas.展开更多
In their recent paper Pereira et al.(2025)claim that validation is overlooked in mapping and modelling of ecosystem services(ES).They state that“many studies lack critical evaluation of the results and no validation ...In their recent paper Pereira et al.(2025)claim that validation is overlooked in mapping and modelling of ecosystem services(ES).They state that“many studies lack critical evaluation of the results and no validation is provided”and that“the validation step is largely overlooked”.This assertion may have been true several years ago,for example,when Ochoa and Urbina-Cardona(2017)made a similar observation.However,there has been much work on ES model validation over the last decade.展开更多
This paper presents an efficient model reduction technique for linear time-varying systems based on shifted Legendre polynomials.The approach constructs approximate low-rank decomposition factors of finite-time Gramia...This paper presents an efficient model reduction technique for linear time-varying systems based on shifted Legendre polynomials.The approach constructs approximate low-rank decomposition factors of finite-time Gramians directly from the expansion coefficients of impulse responses.Leveraging these factors,we develop two model reduction algorithms that integrate the low-rank square root method with dominant subspace projection.Our method is computationally efficient and flexible,requiring only a few matrix-vector operations and a singular value decomposition of a low-dimensional matrix,thereby avoiding the need to solve differential Lyapunov equations.Numerical experiments confirm the effectiveness of the proposed approach.展开更多
In recent years,there has been an increasing need for climate information across diverse sectors of society.This demand has arisen from the necessity to adapt to and mitigate the impacts of climate variability and cha...In recent years,there has been an increasing need for climate information across diverse sectors of society.This demand has arisen from the necessity to adapt to and mitigate the impacts of climate variability and change.Likewise,this period has seen a significant increase in our understanding of the physical processes and mechanisms that drive precipitation and its variability across different regions of Africa.By leveraging a large volume of climate model outputs,numerous studies have investigated the model representation of African precipitation as well as underlying physical processes.These studies have assessed whether the physical processes are well depicted and whether the models are fit for informing mitigation and adaptation strategies.This paper provides a review of the progress in precipitation simulation overAfrica in state-of-the-science climate models and discusses the major issues and challenges that remain.展开更多
文摘Existing methods for tracing water pollution sources typically integrate three-dimensional excitationemission matrix(3D-EEM)fluorescence spectroscopy with similarity-based matching algorithms.However,these approaches exhibit high error rates in borderline cases and necessitate expert manual review,which limits scalability and introduces inconsistencies between algorithmic outputs and expert judgment.To address these limitations,we propose a large vision-language model(VLM)designed as an“expert agent”to automatically refine similarity scores,ensuring alignment with expert decisions and overcoming key application bottlenecks.The model consists of two core components:(1)rule-based similarity calculation module generate initial spectral similarity scores,and(2)pre-trained large vision-language model fine-tuned via supervised learning and reinforcement learning with human feedback(RLHF)to emulate expert assessments.To facilitate training and evaluation,we introduce two expert-annotated datasets,Spec1k and SpecReason,which capture both quantitative corrections and qualitative reasoning patterns,allowing the model to emulate expert decision-making processes.Experimental results demonstrate that our method achieves 81.45%source attribution accuracy,38.24%higher than rule-based and machine learning baselines.Real-world deployment further validates its effectiveness.
文摘In recent years,large vision-language models(VLMs)have achieved significant breakthroughs in cross-modal understanding and generation.However,the safety issues arising from their multimodal interactions become prominent.VLMs are vulnerable to jailbreak attacks,where attackers craft carefully designed prompts to bypass safety mechanisms,leading them to generate harmful content.To address this,we investigate the alignment between visual inputs and task execution,uncovering locality defects and attention biases in VLMs.Based on these findings,we propose VOTI,a novel jailbreak framework leveraging visual obfuscation and task induction.VOTI subtly embeds malicious keywords within neutral image layouts to evade detection,and breaks down harmful queries into a sequence of subtasks.This approach disperses malicious intent across modalities,exploiting VLMs’over-reliance on local visual cues and their fragility in multi-step reasoning to bypass global safety mechanisms.Implemented as an automated framework,VOTI integrates large language models as red-team assistants to generate and iteratively optimize jailbreak strategies.Extensive experiments across seven mainstream VLMs demonstrate VOTI’s effectiveness,achieving a 73.46%attack success rate on GPT-4o-mini.These results reveal critical vulnerabilities in VLMs,highlighting the urgent need for improving robust defenses and multimodal alignment.
基金supported by the Zhejiang Provincial Natural Science Foundation of China(No.LQ23F030001)the National Natural Science Foundation of China(No.62406280)+5 种基金the Autism Research Special Fund of Zhejiang Foundation for Disabled Persons(No.2023008)the Liaoning Province Higher Education Innovative Talents Program Support Project(No.LR2019058)the Liaoning Province Joint Open Fund for Key Scientific and Technological Innovation Bases(No.2021-KF-12-05)the Central Guidance on Local Science and Technology Development Fund of Liaoning Province(No.2023JH6/100100066)the Key Laboratory for Biomedical Engineering of Ministry of Education,Zhejiang University,Chinain part by the Open Research Fund of the State Key Laboratory of Cognitive Neuroscience and Learning.
文摘Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions.Existing methods can be categorized into motion-level,event-level,and story-level ones based on spatiotemporal granularity.However,single-modal approaches struggle to capture complex behavioral semantics and human factors.Therefore,in recent years,vision-language models(VLMs)have been introduced into this field,providing new research perspectives for VAR.In this paper,we systematically review spatiotemporal hierarchical methods in VAR and explore how the introduction of large models has advanced the field.Additionally,we propose the concept of“Factor”to identify and integrate key information from both visual and textual modalities,enhancing multimodal alignment.We also summarize various multimodal alignment methods and provide in-depth analysis and insights into future research directions.
基金supported by the Shanghai Municipal Science and Technology Major Project(No.2021SHZDZX0100)the National Natural Science Foundation of China(No.62088101)+1 种基金the Fundamental Research Funds for the Central Universities(No.22120220642)the Opening Project of the State Key Laboratory of Autonomous Intelligent Unmanned Systems(No.ZZKF2025-2-3).
文摘This paper investigates the potential of Vision-Language Models(VLMs)to enhance Human–Vehicle Interaction(HVI)in Autonomous Driving(AD)scenarios,particularly in interactions between vehicles and other traffic participants,with a focus on rationality and safety in external HVI.Leveraging recent advancements in large language models,VLMs demonstrate remarkable capabilities in understanding real-world contexts and generating significant interest in HVI applications.This paper provides an overview of AD,HVI,and VLMs,along with the historical context of large language model applications in HVI.The HVI discussed herein involves dynamic game processes encompassing perception and decision-making between vehicles and traffic participants,such as pedestrians.Furthermore,we examine the perceptual challenges associated with applying VLMs to HVI and compile relevant datasets.This research fills a gap in the existing literature by systematically analyzing the current status,challenges,and future opportunities of VLM applications in HVI.To advance VLM integration in AD,various implementation strategies are discussed.The findings highlight the potential of VLMs to transform HVI in AD,improving both passenger experience and driving safety.Overall,this study contributes to a comprehensive understanding of VLM applications in HVI and provides insights to guide future research and development.
基金Research Institute for Advanced Manufacturing(RIAM)of The Hong Kong Polytechnic University(1-CDJT)Intra-Faculty Interdisciplinary Project 2023/24(1-WZ4N)+6 种基金Research Committee of The Hong Kong Polytechnic UniversityState Key Laboratory of Intelligent Manufacturing Equipment and Technology,Huazhong University of Science and Technology(IMETKF2024010)Guangdong-Hong Kong Technology Cooperation Funding Scheme(GHX/075/22GD)Innovation and Technology Commission(ITC)COMAC International Collaborative Research Project(COMAC-SFGS-2023-3148)General Research Fund from the Research Grants Council of the Hong Kong Special Administrative Region,China(Project Nos.PolyU15210222 and PolyU15206723)Open access funding provided by the Hong Kong Polytechnic University.
文摘human-robot collaboration(HRC)is set to transform the manufacturing paradigm by leveraging the strengths of human flexibility and robot precision.The recent breakthrough of Large Language Models(LLMs)and Vision-Language Models(VLMs)has motivated the preliminary explorations and adoptions of these models in the smart manufacturing field.However,despite the considerable amount of effort,existing research mainly focused on individual components without a comprehensive perspective to address the full potential of VLMs,especially for HRC in smart manufacturing scenarios.To fill the gap,this work offers a systematic review of the latest advance-ments and applications of VLMs in HRC for smart manu-facturing,which covers the fundamental architectures and pretraining methodologies of LLMs and VLMs,their applications in robotic task planning,navigation,and manipulation,and role in enhancing human-robot skill transfer through multimodal data integration.Lastly,the paper discusses current limitations and future research directions in VLM-based HRC,highlighting the trend in fully realizing the potential of these technologies for smart manufacturing.
文摘In recent years,Vision-Language Models(VLMs)have emerged as a significant breakthrough in multimodal learning,demonstrating remarkable progress in tasks such as image-text alignment,image generation,and semantic reasoning.This paper systematically reviews current VLM pretraining methodologies,including contrastive learning and generative paradigms,while providing an in-depth analysis of efficient transfer learning strategies such as prompt tuning,LoRA,and adapter modules.Through representative models like CLIP,BLIP,and GIT,we examine their practical applications in visual grounding,imagetext retrieval,visual question answering,affective computing,and embodied AI.Furthermore,we identify persistent challenges in fine-grained semantic modeling,cross-modal reasoning,and cross-lingual transfer.Finally,we envision future trends in unified architectures,multimodal reinforcement learning,and domain adaptation,aiming to provide systematic reference and technical insights for subsequent research.
基金The Natural Science Foundation of Hebei Province(F2024501044).
文摘The application of visual-language large models in the field of medical health has gradually become a research focus.The models combine the capability for image understanding and natural language processing,and can simultaneously process multi-modality data such as medical images and medical reports.These models can not only recognize images,but also understand the semantic relationship between images and texts,effectively realize the integration of medical information,and provide strong support for clinical decision-making and disease diagnosis.The visual-language large model has good performance for specific medical tasks,and also shows strong potential and high intelligence in the general task models.This paper provides a comprehensive review of the visual-language large model in the field of medical health.Specifically,this paper first introduces the basic theoretical basis and technical principles.Then,this paper introduces the specific application scenarios in the field of medical health,including modality fusion,semi-supervised learning,weakly supervised learning,unsupervised learning,cross-domain model and general models.Finally,the challenges including insufficient data,interpretability,and practical deployment are discussed.According to the existing challenges,four potential future development directions are given.
文摘In multimodal learning, Vision-Language Models (VLMs) have become a critical research focus, enabling the integration of textual and visual data. These models have shown significant promise across various natural language processing tasks, such as visual question answering and computer vision applications, including image captioning and image-text retrieval, highlighting their adaptability for complex, multimodal datasets. In this work, we review the landscape of Bootstrapping Language-Image Pre-training (BLIP) and other VLM techniques. A comparative analysis is conducted to assess VLMs’ strengths, limitations, and applicability across tasks while examining challenges such as scalability, data quality, and fine-tuning complexities. The work concludes by outlining potential future directions in VLM research, focusing on enhancing model interpretability, addressing ethical implications, and advancing multimodal integration in real-world applications.
基金supported by the National Natural Science Foundation of China(No.62176169)the Fundamental Research Funds for the Central Universities(Nankai University,070-63243150).
文摘The advent of large vision-language models(LVLMs)represents a remarkable advance in the quest for artificial general intelligence.However,the models’effectiveness in both specialized and general tasks warrants further investigation.This paper endeavors to evaluate the competency of popular LVLMs in specialized and general tasks,respectively,aiming to offer a comprehensive understanding of these novel models.To gauge their effectiveness in specialized tasks,we employ six challenging tasks in three different application scenarios:natural,healthcare,and industrial.These six tasks include salient/camouflaged/transparent object detection,as well as polyp detection,skin lesion detection,and industrial anomaly detection.We examine the performance of three recent open-source LVLMs,including MiniGPT-v2,LLaVA-1.5,and Shikra,on both visual recognition and localization in these tasks.Moreover,we conduct empirical investigations utilizing the aforementioned LVLMs together with GPT-4V,assessing their multi-modal understanding capabilities in general tasks including object counting,absurd question answering,affordance reasoning,attribute recognition,and spatial relation reasoning.Our investigations reveal that these LVLMs demonstrate limited proficiency not only in specialized tasks but also in general tasks.We delve deep into this inadequacy and uncover several potential factors,including limited cognition in specialized tasks,object hallucination,text-to-image interference,and decreased robustness in complex problems.We hope that this study can provide useful insights for the future development of LVLMs,helping researchers improve LVLMs for both general and specialized applications.
基金supported in part by the National Natural Science Foundation of China,No.62101136Shanghai Sailing Program,No.21YF1402800National Institutes of Health,Nos.R01CA237267,R01HL151561,R01EB031102,and R01EB032716.
文摘Large language models(LLMs),such as ChatGPT,have demonstrated impressive capabilities in various tasks and attracted increasing interest as a natural language interface across many domains.Recently,large vision-language models(VLMs)that learn rich vision–language correlation from image–text pairs,like BLIP-2 and GPT-4,have been intensively investigated.However,despite these developments,the application of LLMs and VLMs in image quality assessment(IQA),particularly in medical imaging,remains unexplored.This is valuable for objective performance evaluation and potential supplement or even replacement of radiologists’opinions.To this end,this study intro-duces IQAGPT,an innovative computed tomography(CT)IQA system that integrates image-quality captioning VLM with ChatGPT to generate quality scores and textual reports.First,a CT-IQA dataset comprising 1,000 CT slices with diverse quality levels is professionally annotated and compiled for training and evaluation.To better leverage the capabilities of LLMs,the annotated quality scores are converted into semantically rich text descriptions using a prompt template.Second,the image-quality captioning VLM is fine-tuned on the CT-IQA dataset to generate qual-ity descriptions.The captioning model fuses image and text features through cross-modal attention.Third,based on the quality descriptions,users verbally request ChatGPT to rate image-quality scores or produce radiological qual-ity reports.Results demonstrate the feasibility of assessing image quality using LLMs.The proposed IQAGPT outper-formed GPT-4 and CLIP-IQA,as well as multitask classification and regression models that solely rely on images.
基金supported by the National Natural Science Foundation of China (61702528,61806212)。
文摘In the field of satellite imagery, remote sensing image captioning(RSIC) is a hot topic with the challenge of overfitting and difficulty of image and text alignment. To address these issues, this paper proposes a vision-language aligning paradigm for RSIC to jointly represent vision and language. First, a new RSIC dataset DIOR-Captions is built for augmenting object detection in optical remote(DIOR) sensing images dataset with manually annotated Chinese and English contents. Second, a Vision-Language aligning model with Cross-modal Attention(VLCA) is presented to generate accurate and abundant bilingual descriptions for remote sensing images. Third, a crossmodal learning network is introduced to address the problem of visual-lingual alignment. Notably, VLCA is also applied to end-toend Chinese captions generation by using the pre-training language model of Chinese. The experiments are carried out with various baselines to validate VLCA on the proposed dataset. The results demonstrate that the proposed algorithm is more descriptive and informative than existing algorithms in producing captions.
基金supported by the Central Government Guiding Local Science and Technology Development Fund Project(No.2024SZY0343)the Joint Research Program for Ecological Conservation and High Quality Development of the Yellow River Basin(No.2022-YRUC-01-050205)+2 种基金the Higher Education Scientific Research Project of Inner Mongolia Autonomous Region(No.NJZZ23078)the project of Inner Mongolia"Prairie Talents"Engineering Innovation Entrepreneurship Talent Team,the Major Projects of Erdos Science and Technology(No.2022EEDSKJZDZX015)the Innovation Team of the Inner Mongolia Academy of Science and Technology(No.CXTD2023-01-016).
文摘Rural domestic sewage treatment is critical for environmental protection.This study defines the spatial pattern of villages from the perspective of rural sewage treatment and develops an integrated decision-making system to propose a sewage treatment mode and scheme suitable for local conditions.By considering the village spatial layout and terrain factors,a decision tree model of residential density and terrain type was constructed with accuracies of 76.47%and 96.00%,respectively.Combined with binary classification probability unit regression,an appropriate sewage treatment mode for the village was determined with 87.00%accuracy.The Analytic Hierarchy Process(AHP),combined with the Technique for Order Preference(TOPSIS)by Similarity to an Ideal Solution model,formed the basis for optimal treatment process selection under different emission standards.Verification was conducted in 542 villages across three counties of the Inner Mongolia Autonomous Region,focusing on the standard effluent effect(0.3773),low investment cost(0.3196),and high standard effluent effect(0.5115)to determine the best treatment process for the same emission standard under different needs.The annual environmental and carbon emission benefits of sewage treatment in these villages were estimated.This model matches village density,geographic feature,and social development level,and provides scientific support and a theoretical basis for rural sewage treatment decision-making.
基金National Key Scientific and Technological Infrastructure project “Earth System Science Numerical Simulator Facility” (EarthLab)supported by the National Natural Science Foundation of China (Grant No. 92044302)the National Key Research Development Program of China (Grant No. 2022YFC3700703)
文摘We present a comprehensive description and benchmark evaluation of the global–regional chemical transport model called the Emission and Atmospheric Processes Integrated and Coupled Community(EPICC)model.The framework incorporates(1)grid configuration,(2)transport dynamics,(3)chemical mechanisms,(4)aerosol processes,(5)wet/dry deposition parameterizations,and(6)heterogeneous chemistry treatments associated with sulfate,nitrous acid(HONO)chemistry,and aerosol/cloud–photolysis interactions(APIs/CPIs).Openly shared with the atmospheric research community,the model facilitates integration of advanced physicochemical schemes to enhance simulation accuracy.Globally,the model demonstrates realistic representations of ozone(O_(3))and aerosol optical depth.The EPICC model generally demonstrates robust performance in simulating regional concentrations of O_(3) and PM_(2.5)(and its components)in China.It successfully captures vertical profiles of both global and regional O_(3).Notably,the model mitigates frequently reported sulfate underestimations in highly industrialized regions of China.The model accurately captures two regional severe pollution episodes observed in eastern China(January/June 2021).Sensitivity experiments highlight the critical roles of heterogeneous chemical mechanisms associated with sulfate,HONO chemistry,APIs,and CPIs in capturing PM_(2.5) and O_(3) concentrations in China.Improved sulfate mechanisms result in an increase of approximately 32.4%(2.8μg m^(−3))in simulated winter sulfate concentrations when observations exceed 10μg m^(−3).Enhanced HONO elevates winter O_(3) and PM_(2.5) by≤20 and≤10μg m^(−3),respectively.Overall,CPIs dominate over APIs in improving O_(3) and PM_(2.5) simulations across China.Locally,APIs mitigate PM_(2.5) and O_(3) discrepancies in the Sichuan Basin.Seasonal cloud–chemistry coupling explains the weaker impact of PM_(2.5) in summer.
文摘Model evaluation using benchmark datasets is an important method to measure the capability of large language models(LLMs)in specific domains,and it is mainly used to assess the knowledge and reasoning abilities of LLMs.Therefore,in order to better assess the capability of LLMs in the agricultural domain,Agri-Eval was proposed as a benchmark for assessing the knowledge and reasoning ability of LLMs in agriculture.The assessment dataset used in Agri-Eval covered seven major disciplines in the agricultural domain:crop science,horticulture,plant protection,animal husbandry,forest science,aquaculture science,and grass science,and contained a total of 2283 questions.Among domestic general-purpose LLMs,DeepSeek R1 performed best with an accuracy rate of 75.49%.In the realm of international general-purpose LLMs,Gemini 2.0 pro exp 0205 standed out as the top performer,achieving an accuracy rate of 74.28%.As an LLMs in agriculture vertical,Shennong V2.0 outperformed all the LLMs in China,and the answer accuracy rate of agricultural knowledge exceeded that of all the existing general-purpose LLMs.The launch of Agri-Eval helped the LLM developers to comprehensively evaluate the model's capability in the field of agriculture through a variety of tasks and tests to promote the development of the LLMs in the field of agriculture.
文摘We present a novel framework,CLIPSP,and a novel adaptive prompt method to leverage pre-trained knowledge from CLIP for scene parsing.Our approach addresses the limitations of DenseCLIP,which demonstrates the superior image segmentation provided by CLIP pre-trained models over ImageNet pre-trained models,but struggles with rough pixel-text score maps for complex scene parsing.We argue that,as they contain all textual information in a dataset,the pixel-text score maps,i.e.,dense prompts,are inevitably mixed with noise.To overcome this challenge,we propose a two-step method.Firstly,we extract visual and language features and perform multi-label classification to identify the most likely categories in the input images.Secondly,based on the top-k categories and confidence scores,our method generates scene tokens which can be treated as adaptive prompts for implicit modeling of scenes,and incorporates them into the visual features fed into the decoder for segmentation.Our method imposes a constraint on prompts and suppresses the probability of irrelevant categories appearing in the scene parsing results.Our method achieves competitive performance,limited by the available visual-language pre-trained models.Our CLIP-SP performs 1.14%better(in terms of mIoU)than DenseCLIP on ADE20K,using a ResNet-50 backbone.
基金Supported by the National Natural Science Foundation of China(12261018)Universities Key Laboratory of Mathematical Modeling and Data Mining in Guizhou Province(2023013)。
文摘In this paper,we establish and study a single-species logistic model with impulsive age-selective harvesting.First,we prove the ultimate boundedness of the solutions of the system.Then,we obtain conditions for the asymptotic stability of the trivial solution and the positive periodic solution.Finally,numerical simulations are presented to validate our results.Our results show that age-selective harvesting is more conducive to sustainable population survival than non-age-selective harvesting.
文摘The proliferation of high-dimensional data and the widespread use of complex models present central challenges in contemporary statistics and data science.Dimension reduction and model checking,as two foundational pillars supporting scientific inference and data-driven decisionmaking,have evolved through the collective wisdom of generations of statisticians.This special issue,titled"Recent Developments in Dimension Reduction and Model Checking for regressions",not only aims to showcase cutting-edge advances in the field but also carries a distinct sense of academic homage to honor the groundbreaking and enduring contributions of Professor Lixing Zhu,a leading scholar whose work has profoundly shaped both areas.
文摘In their recent paper Pereira et al.(2025)claim that validation is overlooked in mapping and modelling of ecosystem services(ES).They state that“many studies lack critical evaluation of the results and no validation is provided”and that“the validation step is largely overlooked”.This assertion may have been true several years ago,for example,when Ochoa and Urbina-Cardona(2017)made a similar observation.However,there has been much work on ES model validation over the last decade.
文摘This paper presents an efficient model reduction technique for linear time-varying systems based on shifted Legendre polynomials.The approach constructs approximate low-rank decomposition factors of finite-time Gramians directly from the expansion coefficients of impulse responses.Leveraging these factors,we develop two model reduction algorithms that integrate the low-rank square root method with dominant subspace projection.Our method is computationally efficient and flexible,requiring only a few matrix-vector operations and a singular value decomposition of a low-dimensional matrix,thereby avoiding the need to solve differential Lyapunov equations.Numerical experiments confirm the effectiveness of the proposed approach.
基金the World Climate Research Programme(WCRP),Climate Variability and Predictability(CLIVAR),and Global Energy and Water Exchanges(GEWEX)for facilitating the coordination of African monsoon researchsupport from the Center for Earth System Modeling,Analysis,and Data at the Pennsylvania State Universitythe support of the Office of Science of the U.S.Department of Energy Biological and Environmental Research as part of the Regional&Global Model Analysis(RGMA)program area。
文摘In recent years,there has been an increasing need for climate information across diverse sectors of society.This demand has arisen from the necessity to adapt to and mitigate the impacts of climate variability and change.Likewise,this period has seen a significant increase in our understanding of the physical processes and mechanisms that drive precipitation and its variability across different regions of Africa.By leveraging a large volume of climate model outputs,numerous studies have investigated the model representation of African precipitation as well as underlying physical processes.These studies have assessed whether the physical processes are well depicted and whether the models are fit for informing mitigation and adaptation strategies.This paper provides a review of the progress in precipitation simulation overAfrica in state-of-the-science climate models and discusses the major issues and challenges that remain.
