集成多模态大模型的施工安全隐患识别

Multimodal large model-based approach for construction safety hazard recognition

导出

摘要为提升施工场景中安全隐患的自动识别和安全管理水平,构建一个集成多模态大模型的施工安全隐患识别模型,进而构成其核心组件——多模态安全隐患识别模型LLaVA-CS(用于施工场景(Construction Site,CS)下的多模态视觉-文本大语言模型(LLaVA));该系统将图像(施工现场照片)与安全操作规程(工人行为描述)相结合,利用多模态学习和深度学习技术,实时监控和分析施工现场;为支持系统的有效运行,构建一个涵盖不同光照、遮挡和多人场景等复杂条件的多模态数据集,弥补现有公开数据集的空白。结果表明:通过对LLaVA-1.5模型进行提示调优,LLaVA-CS模型能有效融合视觉与文本信息,提升安全隐患识别的精度和可解释性。集成该模型的施工安全隐患识别方法在多个实际施工项目中识别准确率达到0.7222,能够实时生成详细的解释文本,帮助管理人员快速理解安全隐患的具体情境,增强安全管理的决策支持。将多模态大模型应用于施工安全管理系统,有助于提供实时、可解释的安全监控解决方案。 In order to enhance the automatic recognition of safety hazards and improve safety management in construction scenarios,a multimodal large-model-based method for construction safety hazard recognition was proposed and its core component—the multimodal safety hazard recognition model,LLaVA(Large Language and Vision Assistant)-CS(Construction Site),was implemented.The system integrated images(construction site photos)with safety operating procedures(worker behavior descriptions),leveraging multimodal learning and deep learning technologies to perform real-time monitoring and analysis of construction sites.To support the system's effective operation,a multimodal dataset covering complex conditions such as varying lighting,occlusions,and multi-person scenarios was constructed,addressing the gaps in existing public datasets.Through prompt tuning of the LLaVA-1.5 model,the LLaVA-CS model effectively integrated visual and textual information,enhancing the accuracy and interpretability of safety hazard recognition.Experimental results show that this method achieves an accuracy of 0.7222 in multiple real-world construction projects,generating detailed explanatory texts in real time to help managers quickly understand specific safety hazard contexts,thereby improving decision-making in safety management.This study innovatively applies multimodal large models to construction safety management systems,providing real-time,interpretable safety monitoring solutions and offering new technical support and optimization directions for construction safety management.

作者安思齐蔡昂林马子程朱宝岩 AN Siqi;CAI Anglin;MA Zicheng;ZHU Baoyan(School of Safety Science and Engineering,Liaoning Technical University,Huludao Liaoning 125105,China;School of Science,China University of Mining and Technology-Beijing,Beijing 100083,China)

机构地区辽宁工程技术大学安全科学与工程学院中国矿业大学(北京)理学院

出处《中国安全科学学报》北大核心 2025年第9期185-192,共8页 China Safety Science Journal

关键词多模态大模型施工安全隐患复杂施工场景安全管理提示调优 multimodal large model construction safety hazards complex construction scenarios accident prevention tips for tuning

分类号 X948 [环境科学与工程—安全科学]