摘要
多模态情感分析旨在从文本、音频和视觉等多种模态信息中检测出更准确的情感表达。以往的研究通过图神经网络来捕获跨模态和跨时间的节点情感交互,从而获得高度表达的情感信息。但图神经网络只能实现二元信息交互,这限制了对模态间复杂情感交互信息的利用,多模态数据中更需要挖掘这种潜在的情感交互信息。因此,提出了一种基于跨模态超图神经网络的多模态情感分析框架,利用超图结构可以连接多个节点的特性,充分利用模态内和模态间的复杂情感交互信息,以挖掘数据间更深层次的情感表征。此外,提出了一种超图自适应模块来优化学习原始超图的结构。超图自适应网络通过点边交叉注意力、超边采样和节点采样来发现潜在的隐式连接,并修剪冗余的超边以及无关的事件节点,对超图结构进行更新与优化。相对于初始结构,更新后的超图结构能够更准确、更完整地表述数据间的潜在情感关联性,以达到更好的情感分类效果。最后,在两个公开的CMU-MOSI和CMU-MOSEI数据集上进行了广泛的实验,结果表明所提框架相对于其他先进算法在多个性能指标上提升了1%~6%。
Sentiment expressions are multimodal,and more accurate emotions can be derived through multiple modalities such as verbal,audio,and visual.Studying the interactions among modalities can effectively improve the accuracy of multimodal sentiment analysis.Previous studies have used graph models to capture rich interactions across modalities and time to obtain highly expressive and fine-grained sequence representations,but there is a greater need to tap into the expression of higher-order information in multimodal data,which can only be achieved on a one-to-one basis in graph neural networks,which restricts the utilisation of the interactions of higher-order information.This paper explores the application of hypergraph neural networks in multimodal sentiment analysis,where the hypergraph structure can connect two or more nodes to make full use of intra-and inter-modal higher order information and to achieve the interaction of higher-order information between data.Furthermore,this paper proposes a hypergraph adaptive module to optimise the structure of the original hypergraph,where the hypergraph adaptive network is designed to detect potential hidden information by means of point-edge cross-attention,hyperedge sampling and event node sampling to discover potential implicit connections and prune redundant hyperedges as well as irrelevant event nodes to update and optimise the hypergraph structure,the updated hypergraph structure represents the higher-order correlations of the data more accurately and completely than the initial structure.Extensive experiments on two publicly available datasets show that the proposed framework improves 1%to 6%in several performance metrics over other state-of-the-art algorithms on the CMU-MOSI and CMU-MOSEI datasets.
作者
蒋昆
赵征鹏
普园媛
黄健
谷金晶
徐丹
JIANG Kun;ZHAO Zhengpeng;PU Yuanyuan;HUANG Jian;GU Jinjing;XU Dan(School of Information Science and Engineering,Yunnan University,Kunming 650500,China;Internet of Things Technology and Application Key Laboratory of Universities in Yunnan,Kunming 650500,China)
出处
《计算机科学》
北大核心
2025年第7期210-217,共8页
Computer Science
基金
国家自然科学基金(61271361,61761046,62162068,52102382,62362070)
云南省科技厅应用基础研究计划重点项目(202001BB050043,202401AS070149)
云南省科技重大专项(202302AF080006)
研究生科研创新项目(KC-23236053)。
关键词
多模态情感分析
超图神经网络
超图优化
自适应网络
点边信息融合
Multimodal sentiment analysis
Hypergraph neural networks
Hypergraph optimisation
Adaptive networks
Node-edge information fusion
