摘要
图文多模态机器学习结合图片和文本数据,可以提高工业设备识别的准确性。针对现有工业设备识别算法均基于图像数据构建,未能充分利用工业领域普遍存在的文本数据这一问题,提出一种基于图文多模态的工业设备识别模型。该模型包括一个全新的图文数据集和多模态识别算法。数据集图像基于真实场景采集,文本通过多模态大模型自动标注。识别算法能够自适应地调节模态权重,融合两种模态信息,帮助模型决策。实验结果表明,相较于图像单模态数据,所提出算法的识别准确率提高17.62%,相较于其他多模态识别算法,所提出算法的准确率提高5.3%,证明了算法的有效性。
Multimodal machine learning,which combines image and text data,is able to improve the accuracy of industrial equipment identification.To address the limitation that existing industrial equipment identification algorithms rely solely on image data and fail to leverage the widely available textual data in industrial settings,a multimodal industrial equipment identification model based on image and text was proposed.This model consists of a novel image-text dataset and a multimodal identification algorithm.The dataset images were collected from real scenarios,and the text was automatically annotated by a multimodal large model.The identification algorithm can adaptively adjust the weight of the modalities,integrate the information of the two modalities,and assist the model in decision-making.Experimental results show that the proposed algorithm can achieve an improvement of 17.62%in identification accuracy compared to singlemodal image data,and outperforms other multimodal recognition algorithms by 5.3%,demonstrating its effectiveness.
作者
时义聪
赵晓丽
陈明轩
章逾越
SHI Yicong;ZHAO Xiaoli;CHEN Mingxuan;ZHANG Yuyue(School of Electronic and Electrical Engineering,Shanghai University of Engineering Science,Shanghai 201620,China)
出处
《上海工程技术大学学报》
2025年第3期320-325,共6页
Journal of Shanghai University of Engineering Science
基金
上海市科委科技创新行动计划(22DZ2205600)。
关键词
多模态机器学习
多模态融合
工业设备识别
multimodal machine learning
multimodal fusion
industrial equipment identification
