The image emotion classification task aims to use the model to automatically predict the emotional response of people when they see the image.Studies have shown that certain local regions are more likely to inspire an...The image emotion classification task aims to use the model to automatically predict the emotional response of people when they see the image.Studies have shown that certain local regions are more likely to inspire an emotional response than the whole image.However,existing methods perform poorly in predicting the details of emotional regions and are prone to overfitting during training due to the small size of the dataset.Therefore,this study proposes an image emotion classification network based on multilayer attentional interaction and adaptive feature aggregation.To perform more accurate emotional region prediction,this study designs a multilayer attentional interaction module.The module calculates spatial attention maps for higher-layer semantic features and fusion features through amultilayer shuffle attention module.Through layer-by-layer up-sampling and gating operations,the higher-layer features guide the lower-layer features to learn,eventually achieving sentiment region prediction at the optimal scale.To complement the important information lost by layer-by-layer fusion,this study not only adds an intra-layer fusion to the multilayer attention interaction module but also designs an adaptive feature aggregation module.The module uses global average pooling to compress spatial information and connect channel information from all layers.Then,the module adaptively generates a set of aggregated weights through two fully connected layers to augment the original features of each layer.Eventually,the semantics and details of the different layers are aggregated through gating operations and residual connectivity to complement the lost information.To reduce overfitting on small datasets,the network is pre-trained on the FI dataset,and further weight fine-tuning is performed on the small dataset.The experimental results on the FI,Twitter I and Emotion ROI(Region of Interest)datasets show that the proposed network exceeds existing image emotion classification methods,with accuracies of 90.27%,84.66%and 84.96%.展开更多
Image emotion classification(IEC)aims to extract the abstract emotions evoked in images.Recently,language-supervised methods such as con-trastive language-image pretraining(CLIP)have demonstrated superior performance ...Image emotion classification(IEC)aims to extract the abstract emotions evoked in images.Recently,language-supervised methods such as con-trastive language-image pretraining(CLIP)have demonstrated superior performance in image under-standing.However,the underexplored task of IEC presents three major challenges:a tremendous training objective gap between pretraining and IEC,shared suboptimal prompts,and invariant prompts for all instances.In this study,we propose a general framework that effectively exploits the language-supervised CLIP method for the IEC task.First,a prompt-tuning method that mimics the pretraining objective of CLIP is introduced,to exploit the rich image and text semantics associated with CLIP.Subsequently,instance-specific prompts are automatically composed,conditioning them on the categories and image content of instances,diversifying the prompts,and thus avoiding suboptimal problems.Evaluations on six widely used affective datasets show that the proposed method significantly outperforms state-of-the-art methods(up to 9.29%accuracy gain on the EmotionROI dataset)on IEC tasks with only a few trained parameters.The code is publicly available at https://github.com/dsn0w/PT-DPC/for research purposes.展开更多
The combination of electroencephalogram (EEG) and functional magnetic resonance imaging(fMRI) is a very attractive aim in neuroscience in order to achieve both high temporal and spatial resolution for the non-invasive...The combination of electroencephalogram (EEG) and functional magnetic resonance imaging(fMRI) is a very attractive aim in neuroscience in order to achieve both high temporal and spatial resolution for the non-invasive study of cognitive brain function. In this paper, we record simultaneous EEG-fMRI of the same subject in emotional processing experiment in order to explore the characteristics of different emotional picture processing, and try to find the difference of the subjects' brain hemisphere while viewing different valence emotional pictures. The late positive potential(LPP) is a reliable electrophysiological index of emotional perception in humans. According to the analysis results, the slow-wave LPP and visual cortical blood oxygen level-dependent (BOLD) signals are both modulated by the rated intensity of picture arousal. The amplitude of the LPP correlate significantly with BOLD intensity in visual cortex, amygdala, temporal area, prefrontal and central areas across picture contents.展开更多
基金This study was supported,in part,by the National Nature Science Foundation of China under Grant 62272236in part,by the Natural Science Foundation of Jiangsu Province under Grant BK20201136,BK20191401.
文摘The image emotion classification task aims to use the model to automatically predict the emotional response of people when they see the image.Studies have shown that certain local regions are more likely to inspire an emotional response than the whole image.However,existing methods perform poorly in predicting the details of emotional regions and are prone to overfitting during training due to the small size of the dataset.Therefore,this study proposes an image emotion classification network based on multilayer attentional interaction and adaptive feature aggregation.To perform more accurate emotional region prediction,this study designs a multilayer attentional interaction module.The module calculates spatial attention maps for higher-layer semantic features and fusion features through amultilayer shuffle attention module.Through layer-by-layer up-sampling and gating operations,the higher-layer features guide the lower-layer features to learn,eventually achieving sentiment region prediction at the optimal scale.To complement the important information lost by layer-by-layer fusion,this study not only adds an intra-layer fusion to the multilayer attention interaction module but also designs an adaptive feature aggregation module.The module uses global average pooling to compress spatial information and connect channel information from all layers.Then,the module adaptively generates a set of aggregated weights through two fully connected layers to augment the original features of each layer.Eventually,the semantics and details of the different layers are aggregated through gating operations and residual connectivity to complement the lost information.To reduce overfitting on small datasets,the network is pre-trained on the FI dataset,and further weight fine-tuning is performed on the small dataset.The experimental results on the FI,Twitter I and Emotion ROI(Region of Interest)datasets show that the proposed network exceeds existing image emotion classification methods,with accuracies of 90.27%,84.66%and 84.96%.
基金supported in part by the National Natural Science Foundation of China under Grant Nos.62106010,61976010,62176011,62236010.
文摘Image emotion classification(IEC)aims to extract the abstract emotions evoked in images.Recently,language-supervised methods such as con-trastive language-image pretraining(CLIP)have demonstrated superior performance in image under-standing.However,the underexplored task of IEC presents three major challenges:a tremendous training objective gap between pretraining and IEC,shared suboptimal prompts,and invariant prompts for all instances.In this study,we propose a general framework that effectively exploits the language-supervised CLIP method for the IEC task.First,a prompt-tuning method that mimics the pretraining objective of CLIP is introduced,to exploit the rich image and text semantics associated with CLIP.Subsequently,instance-specific prompts are automatically composed,conditioning them on the categories and image content of instances,diversifying the prompts,and thus avoiding suboptimal problems.Evaluations on six widely used affective datasets show that the proposed method significantly outperforms state-of-the-art methods(up to 9.29%accuracy gain on the EmotionROI dataset)on IEC tasks with only a few trained parameters.The code is publicly available at https://github.com/dsn0w/PT-DPC/for research purposes.
基金The Open Project of the State Key Laboratory of Robotics and System at Harbin Institute of Technologygrant number:SKLRS-2010-2D-09,SKLRS-2010-MS-10+5 种基金National Natural Science Foundation of Chinagrant number:61201096Natural Science Foundation of Changzhou Citygrant number:CJ20110023Changzhou High-tech Reasearch Key Laboratory Projectgrant number:CM20123006
文摘The combination of electroencephalogram (EEG) and functional magnetic resonance imaging(fMRI) is a very attractive aim in neuroscience in order to achieve both high temporal and spatial resolution for the non-invasive study of cognitive brain function. In this paper, we record simultaneous EEG-fMRI of the same subject in emotional processing experiment in order to explore the characteristics of different emotional picture processing, and try to find the difference of the subjects' brain hemisphere while viewing different valence emotional pictures. The late positive potential(LPP) is a reliable electrophysiological index of emotional perception in humans. According to the analysis results, the slow-wave LPP and visual cortical blood oxygen level-dependent (BOLD) signals are both modulated by the rated intensity of picture arousal. The amplitude of the LPP correlate significantly with BOLD intensity in visual cortex, amygdala, temporal area, prefrontal and central areas across picture contents.
