A multimodal fusion classifier is presented based on neural networks （NNs） learned with hints for automatic spontaneous affect recognition. In case that different channels can provide com- plementary information, fe...A multimodal fusion classifier is presented based on neural networks （NNs） learned with hints for automatic spontaneous affect recognition. In case that different channels can provide com- plementary information, features are utilized from four behavioral cues： frontal-view facial expres- sion, profile-view facial expression, shoulder movement, and vocalization （audio）. NNs are used in both single cue processing and multimodal fusion. Coarse categories and quadrants in the activation- evaluation dimensional space are utilized respectively as the heuristic information （hints） of NNs during training, aiming at recognition of basic emotions. With the aid of hints, the weights in NNs could learn optimal feature groupings and the subtlety and complexity of spontaneous affective states could be better modeled. The proposed method requires low computation effort and reaches high recognition accuracy, even if the training data is insufficient. Experiment results on the Semaine nat- uralistic dataset demonstrate that our method is effective and promising.展开更多
The thermal-based imaging technique has recently attracted the attention of researchers who are interested in the recognition of human affects dueto its ability to measure the facial transient temperature, which is co...The thermal-based imaging technique has recently attracted the attention of researchers who are interested in the recognition of human affects dueto its ability to measure the facial transient temperature, which is correlated withhuman affects and robustness against illumination changes. Therefore, studieshave increasingly used the thermal imaging as a potential and supplemental solution to overcome the challenges of visual (RGB) imaging, such as the variation oflight conditions and revealing original human affect. Moreover, the thermal-basedimaging has shown promising results in the detection of psychophysiological signals, such as pulse rate and respiration rate in a contactless and noninvasive way.This paper presents a brief review on human affects and focuses on the advantages and challenges of the thermal imaging technique. In addition, this paper discusses the stages of thermal-based human affective state recognition, such asdataset type, preprocessing stage, region of interest (ROI), feature descriptors,and classification approaches with a brief performance analysis based on a number of works in the literature. This analysis could help beginners in the thermalimaging and affective recognition domain to explore numerous approaches usedby researchers to construct an affective state system based on thermal imaging.展开更多
In recent years,the increase of psychopathological disorders in the population has become a health emergency,leading to a great effort to understand psychological vulnerability mechanisms.In this scenario,the role of ...In recent years,the increase of psychopathological disorders in the population has become a health emergency,leading to a great effort to understand psychological vulnerability mechanisms.In this scenario,the role of the autonomic nervous system(ANS)has become increasingly important.This study investigated the association between ANS,social skills,and psychopathological functioning in children.As an ANS status proxy,we measured heart rate variability(HRV).Infants admitted to the neonatal intensive care unit of the University Hospital of Padova because of preterm birth or neonatal hypoxic-ischemic encephalopathy were sequentially recruited from January 2011 to June 2013 and followed long-term up to school age in this cross-sectional observational study.We recorded 5 minutes of HRV immediately before measuring performance in social abilities tasks(affect recognition and theory of mind,NEPSY-II)in 50 children(mean age 7.4±1.4 years)with and without risk factors for developing neuropsychiatric disorders due to pre-/perinatal insults without major sequelae.Children also completed extensive cognitive,neuropsychological,and psychosocial assessment.Parents were assessed with psychopathological interviews and a questionnaire(CBCL 6-18).Analysis in a robust Bayesian framework was used to unearth dependencies between HRV,social skills,and psychopathological functioning.Social task scores were associated with HRV components,with high frequency the most consistent.HRV bands were also associated with the psychopathological questionnaire.Only normalized HRV high frequency was able to distinguish impaired children in the affect recognition task.Our data suggest that ANS may be implicated in social cognition both in typical and atypical developmental conditions and that HRV has cross-disease sensitivity.We suggest that HRV parameters may reflect a neurobiological vulnerability to psychopathology.The study was approved by the Ethics Committee of the University Hospital of Padova(Comitato Etico per la Sperimentazione,Azienda Opedaliera di Padova,approval No.1693 P).展开更多
In dimensional affect recognition, the machine learning methods, which are used to model and predict affect, are mostly classification and regression. However, the annotation in the dimensional affect space usually ta...In dimensional affect recognition, the machine learning methods, which are used to model and predict affect, are mostly classification and regression. However, the annotation in the dimensional affect space usually takes the form of a continuous real value which has an ordinal property. The aforementioned methods do not focus on taking advantage of this important information. Therefore, we propose an affective rating ranking framework for affect recognition based on face images in the valence and arousal dimensional space. Our approach can appropriately use the ordinal information among affective ratings which are generated by discretizing continuous annotations.Specifically, we first train a series of basic cost-sensitive binary classifiers, each of which uses all samples relabeled according to the comparison results between corresponding ratings and a given rank of a binary classifier. We obtain the final affective ratings by aggregating the outputs of binary classifiers. By comparing the experimental results with the baseline and deep learning based classification and regression methods on the benchmarking database of the AVEC 2015 Challenge and the selected subset of SEMAINE database, we find that our ordinal ranking method is effective in both arousal and valence dimensions.展开更多
基金Supported by the National Natural Science Foundation of China(60905006)the Basic Research Fund of Beijing Institute ofTechnology(20120842006)
文摘A multimodal fusion classifier is presented based on neural networks （NNs） learned with hints for automatic spontaneous affect recognition. In case that different channels can provide com- plementary information, features are utilized from four behavioral cues： frontal-view facial expres- sion, profile-view facial expression, shoulder movement, and vocalization （audio）. NNs are used in both single cue processing and multimodal fusion. Coarse categories and quadrants in the activation- evaluation dimensional space are utilized respectively as the heuristic information （hints） of NNs during training, aiming at recognition of basic emotions. With the aid of hints, the weights in NNs could learn optimal feature groupings and the subtlety and complexity of spontaneous affective states could be better modeled. The proposed method requires low computation effort and reaches high recognition accuracy, even if the training data is insufficient. Experiment results on the Semaine nat- uralistic dataset demonstrate that our method is effective and promising.
基金funded by the research university grant by Universiti Sains Malaysia[1001.PKOMP.8014001].
文摘The thermal-based imaging technique has recently attracted the attention of researchers who are interested in the recognition of human affects dueto its ability to measure the facial transient temperature, which is correlated withhuman affects and robustness against illumination changes. Therefore, studieshave increasingly used the thermal imaging as a potential and supplemental solution to overcome the challenges of visual (RGB) imaging, such as the variation oflight conditions and revealing original human affect. Moreover, the thermal-basedimaging has shown promising results in the detection of psychophysiological signals, such as pulse rate and respiration rate in a contactless and noninvasive way.This paper presents a brief review on human affects and focuses on the advantages and challenges of the thermal imaging technique. In addition, this paper discusses the stages of thermal-based human affective state recognition, such asdataset type, preprocessing stage, region of interest (ROI), feature descriptors,and classification approaches with a brief performance analysis based on a number of works in the literature. This analysis could help beginners in the thermalimaging and affective recognition domain to explore numerous approaches usedby researchers to construct an affective state system based on thermal imaging.
文摘In recent years,the increase of psychopathological disorders in the population has become a health emergency,leading to a great effort to understand psychological vulnerability mechanisms.In this scenario,the role of the autonomic nervous system(ANS)has become increasingly important.This study investigated the association between ANS,social skills,and psychopathological functioning in children.As an ANS status proxy,we measured heart rate variability(HRV).Infants admitted to the neonatal intensive care unit of the University Hospital of Padova because of preterm birth or neonatal hypoxic-ischemic encephalopathy were sequentially recruited from January 2011 to June 2013 and followed long-term up to school age in this cross-sectional observational study.We recorded 5 minutes of HRV immediately before measuring performance in social abilities tasks(affect recognition and theory of mind,NEPSY-II)in 50 children(mean age 7.4±1.4 years)with and without risk factors for developing neuropsychiatric disorders due to pre-/perinatal insults without major sequelae.Children also completed extensive cognitive,neuropsychological,and psychosocial assessment.Parents were assessed with psychopathological interviews and a questionnaire(CBCL 6-18).Analysis in a robust Bayesian framework was used to unearth dependencies between HRV,social skills,and psychopathological functioning.Social task scores were associated with HRV components,with high frequency the most consistent.HRV bands were also associated with the psychopathological questionnaire.Only normalized HRV high frequency was able to distinguish impaired children in the affect recognition task.Our data suggest that ANS may be implicated in social cognition both in typical and atypical developmental conditions and that HRV has cross-disease sensitivity.We suggest that HRV parameters may reflect a neurobiological vulnerability to psychopathology.The study was approved by the Ethics Committee of the University Hospital of Padova(Comitato Etico per la Sperimentazione,Azienda Opedaliera di Padova,approval No.1693 P).
基金supported by the National Natural Science Foundation of China(Nos.61272211 and 61672267)the Open Project Program of the National Laboratory of Pattern Recognition(No.201700022)+1 种基金the China Postdoctoral Science Foundation(No.2015M570413)and the Innovation Project of Undergraduate Students in Jiangsu University(No.16A235)
文摘In dimensional affect recognition, the machine learning methods, which are used to model and predict affect, are mostly classification and regression. However, the annotation in the dimensional affect space usually takes the form of a continuous real value which has an ordinal property. The aforementioned methods do not focus on taking advantage of this important information. Therefore, we propose an affective rating ranking framework for affect recognition based on face images in the valence and arousal dimensional space. Our approach can appropriately use the ordinal information among affective ratings which are generated by discretizing continuous annotations.Specifically, we first train a series of basic cost-sensitive binary classifiers, each of which uses all samples relabeled according to the comparison results between corresponding ratings and a given rank of a binary classifier. We obtain the final affective ratings by aggregating the outputs of binary classifiers. By comparing the experimental results with the baseline and deep learning based classification and regression methods on the benchmarking database of the AVEC 2015 Challenge and the selected subset of SEMAINE database, we find that our ordinal ranking method is effective in both arousal and valence dimensions.
