摘要
相较于单模态数据,多模态数据提供了更丰富的信息,使得模型能够从不同维度来分析数据的情感。以往的研究没有充分探索和利用文本与图像数据之间的信息关联。针对上述问题,该文提出一种基于多层的模态渐进交互融合(Multi-Layer Modal Progressive Interaction Fusion,MLMPIF)的图文情感分析模型,以充分探索文本和图像数据之间的信息。该模型从文本和图像中提取到的文本特征、形容词名词对特征和图像特征进行多层渐进交互融合,即先从图像中提取的形容词名词对,增强图像语义理解,并通过门控多模态融合组件实现文本和图像模态的融合,最后,模型引入组合注意力互补组件,促进单模态与融合特征的深入交互,生成更全面的多模态情感特征信息。在MVSA-Single和MVSA-Multiple数据集上进行的实验表明,与主流方法相比,该文的方法对于多模态情感检测是有效的。
Compared with uni-modal data,multimodal data provides richer information,facilitating the sentiment analysis from different dimensions.To better utilize the information association between text and image data,this paper proposes a Multi-Layer Modal Progressive Interaction Fusion(MLMPIF)based graphic sentiment analysis model.In this model,text features,adjective-noun pair features and image features are integrated in multiple layers.That is,adjective-noun pairs are extracted from images to enhance image semantic information,and the fusion of text and image modals is realized through gated multimodal fusion components.Finally,the model introduces the combined attention complementary component to promote the deep interaction between single modal and fusion features.Experiments on both MVSA-Single and MVSA-multiple datasets show that the proposed method is effective for multimodal sentiment detection compared with mainstream methods.
作者
程艳
房成兴
姚宇肖
詹勇鑫
俞恭成
陈彦滢
CHENG Yan;FANG Chengxing;YAO Yuxiao;ZHAN Yongxin;YU Gongcheng;CHEN Yanying(School of Software,Jiangxi Normal University,Nanchang,Jiangxi 330022,China;School of Computer Information Engineering,Jiangxi Normal University,Nanchang,Jiangxi 330022,China;School of Digital Industry,Jiangxi Normal University,Shangrao,Jiangxi 334006,China;Key Laboratory of Intelligent Information Processing and Emotional Computing in Jiangxi Province,Jiangxi Normal University,Nanchang,Jiangxi 330022,China)
出处
《中文信息学报》
北大核心
2025年第5期120-128,共9页
Journal of Chinese Information Processing
基金
国家自然科学基金(62167006)
江西省科技创新基地计划——智能信息处理与情感计算江西省重点实验室(20242BCC32021)
江西省主要学科学术和技术带头人培养计划——领军人才项目(20213BCJL22047)
国家社会科学基金(20AXW009)
江西省自然科学基金(20212BAB202017)。
关键词
图文情感分析
形容词名词对
门控多模态融合
graphic sentiment analysis
adjective noun pairs
gated multimodal fusion
