Multimodal emotion recognition has emerged as a key research area for enabling human-centered artificial intelligence,supported by the rapid progress in vision,audio,language,and physiological modeling.Existing approa...Multimodal emotion recognition has emerged as a key research area for enabling human-centered artificial intelligence,supported by the rapid progress in vision,audio,language,and physiological modeling.Existing approaches integrate heterogeneous affective cues through diverse embedding strategies and fusion mechanisms,yet the field remains fragmented due to differences in feature alignment,temporal synchronization,modality reliability,and robustness to noise or missing inputs.This survey provides a comprehensive analysis of MER research from 2021 to 2025,consolidating advances in modality-specific representation learning,cross-modal feature construction,and early,late,and hybrid fusion paradigms.We systematically review visual,acoustic,textual,and sensor-based embeddings,highlighting howpre-trained encoders,self-supervised learning,and large languagemodels have reshaped the representational foundations ofMER.We further categorize fusion strategies by interaction depth and architectural design,examining how attention mechanisms,cross-modal transformers,adaptive gating,and multimodal large language models redefine the integration of affective signals.Finally,we summarize major benchmark datasets and evaluation metrics and discuss emerging challenges related to scalability,generalization,and interpretability.This survey aims to provide a unified perspective onmultimodal fusion for emotion recognition and to guide future research toward more coherent and generalizable multimodal affective intelligence.展开更多
This editorial comments on an article recently published by López del Hoyo et al.The metaverse,hailed as"the successor to the mobile Internet",is undoubtedly one of the most fashionable terms in recent ...This editorial comments on an article recently published by López del Hoyo et al.The metaverse,hailed as"the successor to the mobile Internet",is undoubtedly one of the most fashionable terms in recent years.Although metaverse development is a complex and multifaceted evolutionary process influenced by many factors,it is almost certain that it will significantly impact our lives,including mental health services.Like any other technological advancements,the metaverse era presents a double-edged sword for mental health work,which must clearly understand the needs and transformations of its target audience.In this editorial,our primary focus is to contemplate potential new needs and transformation in mental health work during the metaverse era from the pers-pective of multimodal emotion recognition.展开更多
Multimodal emotion recognition plays a vital role in social media analysis.However,traditional systems face a fundamental challenge:cross-modal semantic conflicts triggered by internet memes lead to a severe negative ...Multimodal emotion recognition plays a vital role in social media analysis.However,traditional systems face a fundamental challenge:cross-modal semantic conflicts triggered by internet memes lead to a severe negative transfer effect-significantly degrading the model's ability to recognise emotions even in regular non-meme content.This study systematically reveals and quantifies this phenomenon,providing critical insights for overcoming the bottleneck of multimodal recognition in meme-rich contexts.To investigate this issue,we constructed a balanced mixed dataset and first empirically confirmed that conventional fusion methods suffer a substantial performance drop under this setting.In response,we propose the Meme-Adaptive Gated Fusion Network(MAG-Net),which is built upon shared Bidirectional Encoder Representations from Transformers(BERT)-Residual Neural Network(ResNet)encoders,incorporates a gated fusion module to dynamically adjust the fusion weights of image and text features,and employs a multi-task learning framework to jointly optimise emotion classification and meme detection.Through feature sharing and the adaptive modulation of fusion ratios via the gating network,our method effectively mitigates crossmodal conflicts.Our experiments demonstrate that on a test set containing 50%memes,the method achieves 56.93%emotion recognition accuracy,outperforming the baseline by 14.65%,while reaching 94.02%meme detection accuracy.Importantly,our framework not only improves overall performance but also restores the model's ability to correctly process regular content,thereby recovering the advantages of multimodal fusion in challenging meme-rich scenarios.展开更多
Multimodal emotion recognition technology leverages the power of deep learning to address advanced visual and emotional tasks.While generic deep networks can handle simple emotion recognition tasks,their generalizatio...Multimodal emotion recognition technology leverages the power of deep learning to address advanced visual and emotional tasks.While generic deep networks can handle simple emotion recognition tasks,their generalization capability in complex and noisy environments,such as multi-scene outdoor settings,remains limited.To overcome these challenges,this paper proposes a novel multimodal emotion recognition framework.First,we develop a robust network architecture based on the T5-small model,designed for dynamic-static fusion in complex scenarios,effectively mitigating the impact of noise.Second,we introduce a dynamic-static cross fusion network(D-SCFN)to enhance the integration and extraction of dynamic and static information,embedding it seamlessly within the T5 framework.Finally,we design and evaluate three distinct multi-task analysis frameworks to explore dependencies among tasks.The experimental results demonstrate that our model significantly outperforms other existing models,showcasing exceptional stability and remarkable adaptability to complex and dynamic scenarios.展开更多
Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition.Existing approaches use directional pairwise attention or a messag...Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition.Existing approaches use directional pairwise attention or a message hub to fuse language,visual,and audio modalities.However,these fusion methods are often quadratic in complexity with respect to the modal sequence length,bring redundant information and are not efficient.In this paper,we propose an efficient neural network to learn modality-fused representations with CB-Transformer(LMR-CBT)for multimodal emotion recognition from unaligned multi-modal sequences.Specifically,we first perform feature extraction for the three modalities respectively to obtain the local structure of the sequences.Then,we design an innovative asymmetric transformer with cross-modal blocks(CB-Transformer)that enables complementary learning of different modalities,mainly divided into local temporal learning,cross-modal feature fusion and global self-attention representations.In addition,we splice the fused features with the original features to classify the emotions of the sequences.Finally,we conduct word-aligned and unaligned experiments on three challenging datasets,IEMOCAP,CMU-MOSI,and CMU-MOSEI.The experimental results show the superiority and efficiency of our proposed method in both settings.Compared with the mainstream methods,our approach reaches the state-of-the-art with a minimum number of parameters.展开更多
The fusion technique is the key to the multimodal emotion recognition task.Recently,cross-modal attention-based fusion methods have demonstrated high performance and strong robustness.However,cross-modal attention suf...The fusion technique is the key to the multimodal emotion recognition task.Recently,cross-modal attention-based fusion methods have demonstrated high performance and strong robustness.However,cross-modal attention suffers from redundant features and does not capture complementary features well.We find that it is not necessary to use the entire information of one modality to reinforce the other during cross-modal interaction,and the features that can reinforce a modality may contain only a part of it.To this end,we design an innovative Transformer-based Adaptive Cross-modal Fusion Network(TACFN).Specifically,for the redundant features,we make one modality perform intra-modal feature selection through a self-attention mechanism,so that the selected features can adaptively and efficiently interact with another modality.To better capture the complementary information between the modalities,we obtain the fused weight vector by splicing and use the weight vector to achieve feature reinforcement of the modalities.We apply TCAFN to the RAVDESS and IEMOCAP datasets.For fair comparison,we use the same unimodal representations to validate the effectiveness of the proposed fusion method.The experimental results show that TACFN brings a significant performance improvement compared to other methods and reaches the state-of-the-art performance.All code and models could be accessed from https://github.com/shuzihuaiyu/TACFN.展开更多
In the realm of human-computer interaction, accurately discerning the user's emotional state during a conversation has become increasingly critical. Multimodal emotion recognition has garnered considerable attenti...In the realm of human-computer interaction, accurately discerning the user's emotional state during a conversation has become increasingly critical. Multimodal emotion recognition has garnered considerable attention.However, the task of multimodal emotion recognition still faces several challenges. Especially, the models are unable to effectively extract multimodal contextual and interaction information, which results in a significant redundancy in the concatenated features representation. To mitigate this issue, a cross-modal fusion network based on graph feature learning(CFNet-GFL) model is proposed. Firstly, the model employs a cross-modal module to integrate multiple feature representations, resulting in more precise feature embedding. This alleviates the issue of information redundancy in the fusion process. Secondly, the model leverages a graph feature learning approach to extract intra-modal contextual preferences and inter-modal information interactions. By capturing the diversity and consistency of multimodal information, the model enhances the ability to understand emotions. Finally, the performance of the CFNet-GFL model is demonstrated by some experiments on the IEMOCAP and MELD datasets. The w-F1 scores show a significant improvement of approximately 2.08% and 1.36% respectively. These findings demonstrate the effectiveness of the model in multimodal emotion recognition.展开更多
With the development of artificial intelligence,emotion recognition has become a hot topic in the field of humancomputer interaction.This paper focuses on the application and optimization of deep convolutional neural ...With the development of artificial intelligence,emotion recognition has become a hot topic in the field of humancomputer interaction.This paper focuses on the application and optimization of deep convolutional neural networks(CNNs)in multimodal emotion recognition.Multimodal emotion recognition involves analyzing data from different sources—such as voice,facial expressions,and text—to more accurately identify and interpret human emotional states.This paper first reviews the basic theories and methods of multimodal data processing,then details the structure and function of deep convolutional neural networks,particularly their advantages in handling various types of data.By innovating and optimizing network structures,loss functions,and training strategies,we have improved the model's accuracy in emotion recognition.Ultimately,experimental results show that the optimized CNN model demonstrates superior performance in multimodal emotion recognition tasks.展开更多
A Tsinghua-developed biometric recognition system, designed to bolster traditional public security identification measures, was highly commended in an appraisal by the Ministry of Education on June 22, 2005.
基金supported by the Institute of Information&Communications Technology Planning&Evaluation grant funded by the Korea government(MSIT)(No.RS-2021-II211341,AI Graduate School Support Program,Chung-Ang University)in part by the Institute of Information and Communications Technology Planning and Evaluation grant funded by the Korea government(MSIT)(Development of Integrated Development Framework that Supports Automatic Neural Network Generation and Deployment Optimized for Runtime Environment,Grant No.2021-0-00766).
文摘Multimodal emotion recognition has emerged as a key research area for enabling human-centered artificial intelligence,supported by the rapid progress in vision,audio,language,and physiological modeling.Existing approaches integrate heterogeneous affective cues through diverse embedding strategies and fusion mechanisms,yet the field remains fragmented due to differences in feature alignment,temporal synchronization,modality reliability,and robustness to noise or missing inputs.This survey provides a comprehensive analysis of MER research from 2021 to 2025,consolidating advances in modality-specific representation learning,cross-modal feature construction,and early,late,and hybrid fusion paradigms.We systematically review visual,acoustic,textual,and sensor-based embeddings,highlighting howpre-trained encoders,self-supervised learning,and large languagemodels have reshaped the representational foundations ofMER.We further categorize fusion strategies by interaction depth and architectural design,examining how attention mechanisms,cross-modal transformers,adaptive gating,and multimodal large language models redefine the integration of affective signals.Finally,we summarize major benchmark datasets and evaluation metrics and discuss emerging challenges related to scalability,generalization,and interpretability.This survey aims to provide a unified perspective onmultimodal fusion for emotion recognition and to guide future research toward more coherent and generalizable multimodal affective intelligence.
基金Supported by Education and Teaching Reform Project of the First Clinical College of Chongqing Medical University,No.CMER202305Natural Science Foundation of Tibet Autonomous Region,No.XZ2024ZR-ZY100(Z).
文摘This editorial comments on an article recently published by López del Hoyo et al.The metaverse,hailed as"the successor to the mobile Internet",is undoubtedly one of the most fashionable terms in recent years.Although metaverse development is a complex and multifaceted evolutionary process influenced by many factors,it is almost certain that it will significantly impact our lives,including mental health services.Like any other technological advancements,the metaverse era presents a double-edged sword for mental health work,which must clearly understand the needs and transformations of its target audience.In this editorial,our primary focus is to contemplate potential new needs and transformation in mental health work during the metaverse era from the pers-pective of multimodal emotion recognition.
文摘Multimodal emotion recognition plays a vital role in social media analysis.However,traditional systems face a fundamental challenge:cross-modal semantic conflicts triggered by internet memes lead to a severe negative transfer effect-significantly degrading the model's ability to recognise emotions even in regular non-meme content.This study systematically reveals and quantifies this phenomenon,providing critical insights for overcoming the bottleneck of multimodal recognition in meme-rich contexts.To investigate this issue,we constructed a balanced mixed dataset and first empirically confirmed that conventional fusion methods suffer a substantial performance drop under this setting.In response,we propose the Meme-Adaptive Gated Fusion Network(MAG-Net),which is built upon shared Bidirectional Encoder Representations from Transformers(BERT)-Residual Neural Network(ResNet)encoders,incorporates a gated fusion module to dynamically adjust the fusion weights of image and text features,and employs a multi-task learning framework to jointly optimise emotion classification and meme detection.Through feature sharing and the adaptive modulation of fusion ratios via the gating network,our method effectively mitigates crossmodal conflicts.Our experiments demonstrate that on a test set containing 50%memes,the method achieves 56.93%emotion recognition accuracy,outperforming the baseline by 14.65%,while reaching 94.02%meme detection accuracy.Importantly,our framework not only improves overall performance but also restores the model's ability to correctly process regular content,thereby recovering the advantages of multimodal fusion in challenging meme-rich scenarios.
基金funded by the Foundation Item:Shaanxi Provincial Technical Innovation Guidance Plan(2023KXJ-279).
文摘Multimodal emotion recognition technology leverages the power of deep learning to address advanced visual and emotional tasks.While generic deep networks can handle simple emotion recognition tasks,their generalization capability in complex and noisy environments,such as multi-scene outdoor settings,remains limited.To overcome these challenges,this paper proposes a novel multimodal emotion recognition framework.First,we develop a robust network architecture based on the T5-small model,designed for dynamic-static fusion in complex scenarios,effectively mitigating the impact of noise.Second,we introduce a dynamic-static cross fusion network(D-SCFN)to enhance the integration and extraction of dynamic and static information,embedding it seamlessly within the T5 framework.Finally,we design and evaluate three distinct multi-task analysis frameworks to explore dependencies among tasks.The experimental results demonstrate that our model significantly outperforms other existing models,showcasing exceptional stability and remarkable adaptability to complex and dynamic scenarios.
基金National Natural Science Foundation of China(Grant No.72293583).
文摘Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition.Existing approaches use directional pairwise attention or a message hub to fuse language,visual,and audio modalities.However,these fusion methods are often quadratic in complexity with respect to the modal sequence length,bring redundant information and are not efficient.In this paper,we propose an efficient neural network to learn modality-fused representations with CB-Transformer(LMR-CBT)for multimodal emotion recognition from unaligned multi-modal sequences.Specifically,we first perform feature extraction for the three modalities respectively to obtain the local structure of the sequences.Then,we design an innovative asymmetric transformer with cross-modal blocks(CB-Transformer)that enables complementary learning of different modalities,mainly divided into local temporal learning,cross-modal feature fusion and global self-attention representations.In addition,we splice the fused features with the original features to classify the emotions of the sequences.Finally,we conduct word-aligned and unaligned experiments on three challenging datasets,IEMOCAP,CMU-MOSI,and CMU-MOSEI.The experimental results show the superiority and efficiency of our proposed method in both settings.Compared with the mainstream methods,our approach reaches the state-of-the-art with a minimum number of parameters.
基金supported by Beijing Key Laboratory of Behavior and Mental Health,Peking University。
文摘The fusion technique is the key to the multimodal emotion recognition task.Recently,cross-modal attention-based fusion methods have demonstrated high performance and strong robustness.However,cross-modal attention suffers from redundant features and does not capture complementary features well.We find that it is not necessary to use the entire information of one modality to reinforce the other during cross-modal interaction,and the features that can reinforce a modality may contain only a part of it.To this end,we design an innovative Transformer-based Adaptive Cross-modal Fusion Network(TACFN).Specifically,for the redundant features,we make one modality perform intra-modal feature selection through a self-attention mechanism,so that the selected features can adaptively and efficiently interact with another modality.To better capture the complementary information between the modalities,we obtain the fused weight vector by splicing and use the weight vector to achieve feature reinforcement of the modalities.We apply TCAFN to the RAVDESS and IEMOCAP datasets.For fair comparison,we use the same unimodal representations to validate the effectiveness of the proposed fusion method.The experimental results show that TACFN brings a significant performance improvement compared to other methods and reaches the state-of-the-art performance.All code and models could be accessed from https://github.com/shuzihuaiyu/TACFN.
基金supported by National Natural Science Foundation of China (62201452)。
文摘In the realm of human-computer interaction, accurately discerning the user's emotional state during a conversation has become increasingly critical. Multimodal emotion recognition has garnered considerable attention.However, the task of multimodal emotion recognition still faces several challenges. Especially, the models are unable to effectively extract multimodal contextual and interaction information, which results in a significant redundancy in the concatenated features representation. To mitigate this issue, a cross-modal fusion network based on graph feature learning(CFNet-GFL) model is proposed. Firstly, the model employs a cross-modal module to integrate multiple feature representations, resulting in more precise feature embedding. This alleviates the issue of information redundancy in the fusion process. Secondly, the model leverages a graph feature learning approach to extract intra-modal contextual preferences and inter-modal information interactions. By capturing the diversity and consistency of multimodal information, the model enhances the ability to understand emotions. Finally, the performance of the CFNet-GFL model is demonstrated by some experiments on the IEMOCAP and MELD datasets. The w-F1 scores show a significant improvement of approximately 2.08% and 1.36% respectively. These findings demonstrate the effectiveness of the model in multimodal emotion recognition.
文摘With the development of artificial intelligence,emotion recognition has become a hot topic in the field of humancomputer interaction.This paper focuses on the application and optimization of deep convolutional neural networks(CNNs)in multimodal emotion recognition.Multimodal emotion recognition involves analyzing data from different sources—such as voice,facial expressions,and text—to more accurately identify and interpret human emotional states.This paper first reviews the basic theories and methods of multimodal data processing,then details the structure and function of deep convolutional neural networks,particularly their advantages in handling various types of data.By innovating and optimizing network structures,loss functions,and training strategies,we have improved the model's accuracy in emotion recognition.Ultimately,experimental results show that the optimized CNN model demonstrates superior performance in multimodal emotion recognition tasks.
文摘A Tsinghua-developed biometric recognition system, designed to bolster traditional public security identification measures, was highly commended in an appraisal by the Ministry of Education on June 22, 2005.
