Artificial intelligence,such as deep learning technology,has advanced the study of facial expression recognition since facial expression carries rich emotional information and is significant for many naturalistic situ...Artificial intelligence,such as deep learning technology,has advanced the study of facial expression recognition since facial expression carries rich emotional information and is significant for many naturalistic situations.To pursue a high facial expression recognition accuracy,the network model of deep learning is generally designed to be very deep while the model’s real-time performance is typically constrained and limited.With MobileNetV3,a lightweight model with a good accuracy,a further study is conducted by adding a basic ResNet module to each of its existing modules and an SSH(Single Stage Headless Face Detector)context module to expand the model’s perceptual field.In this article,the enhanced model named Res-MobileNetV3,could alleviate the subpar of real-time performance and compress the size of large network models,which can process information at a rate of up to 33 frames per second.Although the improved model has been verified to be slightly inferior to the current state-of-the-art method in aspect of accuracy rate on the publically available face expression datasets,it can bring a good balance on accuracy,real-time performance,model size and model complexity in practical applications.展开更多
The dynamics of student engagement and emotional states significantly influence learning outcomes.Positive emotions resulting from successful task completion stand in contrast to negative affective states that arise f...The dynamics of student engagement and emotional states significantly influence learning outcomes.Positive emotions resulting from successful task completion stand in contrast to negative affective states that arise from learning struggles or failures.Effective transitions to engagement occur upon problem resolution,while unresolved issues lead to frustration and subsequent boredom.This study proposes a Convolutional Neural Networks(CNN)based approach utilizing the Multi⁃source Academic Affective Engagement Dataset(MAAED)to categorize facial expressions into boredom,confusion,frustration,and yawning.This method provides an efficient and objective way to assess student engagement by extracting features from facial images.Recognizing and addressing negative affective states,such as confusion and boredom,is fundamental in creating supportive learning environments.Through automated frame extraction and model comparison,this study demonstrates reduced loss values with improving accuracy,showcasing the effectiveness of this method in objectively evaluating student engagement.Monitoring facial engagement with CNN using the MAAED dataset is essential for gaining insights into human behaviour and improving educational experiences.展开更多
The estimation of pain intensity is critical for medical diagnosis and treatment of patients.With the development of image monitoring technology and artificial intelligence,automatic pain assessment based on facial ex...The estimation of pain intensity is critical for medical diagnosis and treatment of patients.With the development of image monitoring technology and artificial intelligence,automatic pain assessment based on facial expression and behavioral analysis shows a potential value in clinical applications.This paper reports a framework of convolutional neural network with global and local attention mechanism(GLA-CNN)for the effective detection of pain intensity at four-level thresholds using facial expression images.GLA-CNN includes two modules,namely global attention network(GANet)and local attention network(LANet).LANet is responsible for extracting representative local patch features of faces,while GANet extracts whole facial features to compensate for the ignored correlative features between patches.In the end,the global correlational and local subtle features are fused for the final estimation of pain intensity.Experiments under the UNBC-McMaster Shoulder Pain database demonstrate that GLA-CNN outperforms other state-of-the-art methods.Additionally,a visualization analysis is conducted to present the feature map of GLA-CNN,intuitively showing that it can extract not only local pain features but also global correlative facial ones.Our study demonstrates that pain assessment based on facial expression is a non-invasive and feasible method,and can be employed as an auxiliary pain assessment tool in clinical practice.展开更多
A novel fuzzy linear discriminant analysis method by the canonical correlation analysis （fuzzy-LDA/CCA）is presented and applied to the facial expression recognition. The fuzzy method is used to evaluate the degree o...A novel fuzzy linear discriminant analysis method by the canonical correlation analysis （fuzzy-LDA/CCA）is presented and applied to the facial expression recognition. The fuzzy method is used to evaluate the degree of the class membership to which each training sample belongs. CCA is then used to establish the relationship between each facial image and the corresponding class membership vector, and the class membership vector of a test image is estimated using this relationship. Moreover, the fuzzy-LDA/CCA method is also generalized to deal with nonlinear discriminant analysis problems via kernel method. The performance of the proposed method is demonstrated using real data.展开更多
It is unknown if the ability of Portuguese in the identification of NimStim data set,which was created in America to provide facial expressions that could be recognized by untrained people,is(or not)similar to the Ame...It is unknown if the ability of Portuguese in the identification of NimStim data set,which was created in America to provide facial expressions that could be recognized by untrained people,is(or not)similar to the Americans.To test this hypothesis the performance of Portuguese in the recognition of Happiness,Surprise,Sadness,Fear,Disgust and Anger NimStim facial expressions was compared with the Americans,but no significant differences were found.In both populations the easiest emotion to identify was Happiness while Fear was the most difficult one.However,with exception for Surprise,Portuguese tend to show a lower accuracy rate for all the emotions studied.Results highlighted some cultural differences.展开更多
Facial expression recognition is a hot topic in computer vision, but it remains challenging due to the feature inconsistency caused by person-specific 'characteristics of facial expressions. To address such a chal...Facial expression recognition is a hot topic in computer vision, but it remains challenging due to the feature inconsistency caused by person-specific 'characteristics of facial expressions. To address such a challenge, and inspired by the recent success of deep identity network (DeepID-Net) for face identification, this paper proposes a novel deep learning based framework for recognising human expressions with facial images. Compared to the existing deep learning methods, our proposed framework, which is based on multi-scale global images and local facial patches, can significantly achieve a better performance on facial expression recognition. Finally, we verify the effectiveness of our proposed framework through experiments on the public benchmarking datasets JAFFE and extended Cohn-Kanade (CK+).展开更多
In expression recognition, feature representation is critical for successful recognition since it contains distinctive information of expressions. In this paper, a new approach for representing facial expression featu...In expression recognition, feature representation is critical for successful recognition since it contains distinctive information of expressions. In this paper, a new approach for representing facial expression features is proposed with its objective to describe features in an effective and efficient way in order to improve the recognition performance. The method combines the facial action coding system(FACS) and 'uniform' local binary patterns(LBP) to represent facial expression features from coarse to fine. The facial feature regions are extracted by active shape models(ASM) based on FACS to obtain the gray-level texture. Then, LBP is used to represent expression features for enhancing the discriminant. A facial expression recognition system is developed based on this feature extraction method by using K nearest neighborhood(K-NN) classifier to recognize facial expressions. Finally, experiments are carried out to evaluate this feature extraction method. The significance of removing the unrelated facial regions and enhancing the discrimination ability of expression features in the recognition process is indicated by the results, in addition to its convenience.展开更多
A facial expression emotion recognition based human-robot interaction(FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize huma...A facial expression emotion recognition based human-robot interaction(FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on2D-Gabor, uniform local binary pattern(LBP) operator, and multiclass extreme learning machine(ELM) classifier is presented,which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios,i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.展开更多
Functional magnetic resonance imaging was used during emotion recognition to identify changes in functional brain activation in 21 first-episode, treatment-naive major depressive disorder patients before and after ant...Functional magnetic resonance imaging was used during emotion recognition to identify changes in functional brain activation in 21 first-episode, treatment-naive major depressive disorder patients before and after antidepressant treatment. Following escitalopram oxalate treatment, patients exhibited decreased activation in bilateral precentral gyrus, bilateral middle frontal gyrus, left middle temporal gyrus, bilateral postcentral gyrus, left cingulate and right parahippocampal gyrus, and increased activation in right superior frontal gyrus, bilateral superior parietal Iobule and left occipital gyrus during sad facial expression recognition. After antidepressant treatment, patients also exhibited decreased activation in the bilateral middle frontal gyrus, bilateral cingulate and right parahippocampal gyrus, and increased activation in the right inferior frontal gyrus, left fusiform gyrus and right precuneus during happy facial expression recognition. Our experimental findings indicate that the limbic-cortical network might be a key target region for antidepressant treatment in major depressive disorder.展开更多
Facial Expression Recognition(FER)has been an interesting area of research in places where there is human-computer interaction.Human psychol-ogy,emotions and behaviors can be analyzed in FER.Classifiers used in FER hav...Facial Expression Recognition(FER)has been an interesting area of research in places where there is human-computer interaction.Human psychol-ogy,emotions and behaviors can be analyzed in FER.Classifiers used in FER have been perfect on normal faces but have been found to be constrained in occluded faces.Recently,Deep Learning Techniques(DLT)have gained popular-ity in applications of real-world problems including recognition of human emo-tions.The human face reflects emotional states and human intentions.An expression is the most natural and powerful way of communicating non-verbally.Systems which form communications between the two are termed Human Machine Interaction(HMI)systems.FER can improve HMI systems as human expressions convey useful information to an observer.This paper proposes a FER scheme called EECNN(Enhanced Convolution Neural Network with Atten-tion mechanism)to recognize seven types of human emotions with satisfying results in its experiments.Proposed EECNN achieved 89.8%accuracy in classi-fying the images.展开更多
For the problems of complex model structure and too many training parameters in facial expression recognition algorithms,we proposed a residual network structure with a multi-headed channel attention(MCA)module.The mi...For the problems of complex model structure and too many training parameters in facial expression recognition algorithms,we proposed a residual network structure with a multi-headed channel attention(MCA)module.The migration learning algorithm is used to pre-train the convolutional layer parameters and mitigate the overfitting caused by the insufficient number of training samples.The designed MCA module is integrated into the ResNet18 backbone network.The attention mechanism highlights important information and suppresses irrelevant information by assigning different coefficients or weights,and the multi-head structure focuses more on the local features of the pictures,which improves the efficiency of facial expression recognition.Experimental results demonstrate that the model proposed in this paper achieves excellent recognition results in Fer2013,CK+and Jaffe datasets,with accuracy rates of 72.7%,98.8%and 93.33%,respectively.展开更多
Local binary pattern(LBP)is an important method for texture feature extraction of facial expression.However,it also has the shortcomings of high dimension,slow feature extraction and noeffective local or global featur...Local binary pattern(LBP)is an important method for texture feature extraction of facial expression.However,it also has the shortcomings of high dimension,slow feature extraction and noeffective local or global features extracted.To solve these problems,a facial expression feature extraction method is proposed based on improved LBP.Firstly,LBP is converted into double local binary pattern(DLBP).Then by combining Taylor expansion(TE)with DLBP,DLBP-TE algorithm is obtained.Finally,the DLBP-TE algorithm combined with extreme learning machine(ELM)is applied in seven kinds of ficial expression images and the corresponding experiments are carried out in Japanese adult female facial expression(JAFFE)database.The results show that the proposed method can significantly improve facial expression recognition rate.展开更多
In computer vision,emotion recognition using facial expression images is considered an important research issue.Deep learning advances in recent years have aided in attaining improved results in this issue.According t...In computer vision,emotion recognition using facial expression images is considered an important research issue.Deep learning advances in recent years have aided in attaining improved results in this issue.According to recent studies,multiple facial expressions may be included in facial photographs representing a particular type of emotion.It is feasible and useful to convert face photos into collections of visual words and carry out global expression recognition.The main contribution of this paper is to propose a facial expression recognitionmodel(FERM)depending on an optimized Support Vector Machine(SVM).To test the performance of the proposed model(FERM),AffectNet is used.AffectNet uses 1250 emotion-related keywords in six different languages to search three major search engines and get over 1,000,000 facial photos online.The FERM is composed of three main phases:(i)the Data preparation phase,(ii)Applying grid search for optimization,and(iii)the categorization phase.Linear discriminant analysis(LDA)is used to categorize the data into eight labels(neutral,happy,sad,surprised,fear,disgust,angry,and contempt).Due to using LDA,the performance of categorization via SVM has been obviously enhanced.Grid search is used to find the optimal values for hyperparameters of SVM(C and gamma).The proposed optimized SVM algorithm has achieved an accuracy of 99%and a 98%F1 score.展开更多
OBJECTIVE: The objective of this study is to summarize and analyze the brain signal patterns of empathy for pain caused by facial expressions of pain utilizing activation likelihood estimation, a meta-analysis method....OBJECTIVE: The objective of this study is to summarize and analyze the brain signal patterns of empathy for pain caused by facial expressions of pain utilizing activation likelihood estimation, a meta-analysis method. DATA SOURCES: Studies concerning the brain mechanism were searched from the Science Citation Index, Science Direct, PubMed, DeepDyve, Cochrane Library, SinoMed, Wanfang, VIP, China National Knowledge Infrastructure, and other databases, such as SpringerLink, AMA, Science Online, Wiley Online, were collected. A time limitation of up to 13 December 2016 was applied to this study. DATA SELECTION: Studies presenting with all of the following criteria were considered for study inclusion: Use of functional magnetic resonance imaging, neutral and pained facial expression stimuli, involvement of adult healthy human participants over 18 years of age, whose empathy ability showed no difference from the healthy adult, a painless basic state, results presented in Talairach or Montreal Neurological Institute coordinates, multiple studies by the same team as long as they used different raw data. OUTCOME MEASURES: Activation likelihood estimation was used to calculate the combined main activated brain regions under the stimulation of pained facial expression. RESULTS: Eight studies were included, containing 178 subjects. Meta-analysis results suggested that the anterior cingulate cortex(BA32), anterior central gyrus(BA44), fusiform gyrus, and insula(BA13) were activated positively as major brain areas under the stimulation of pained facial expression. CONCLUSION: Our study shows that pained facial expression alone, without viewing of painful stimuli, activated brain regions related to pain empathy, further contributing to revealing the brain's mechanisms of pain empathy.展开更多
In this paper, a novel method based on dual-tree complex wavelet transform(DT-CWT) and rotation invariant local binary pattern(LBP) for facial expression recognition is proposed. The quarter sample shift (Q-shift) DT-...In this paper, a novel method based on dual-tree complex wavelet transform(DT-CWT) and rotation invariant local binary pattern(LBP) for facial expression recognition is proposed. The quarter sample shift (Q-shift) DT-CWT can provide a group delay of 1/4 of a sample period, and satisfy the usual 2-band filter bank constraints of no aliasing and perfect reconstruction. To resolve illumination variation in expression verification, low-frequency coefficients produced by DT-CWT are set zeroes, high-frequency coefficients are used for reconstructing the image, and basic LBP histogram is mapped on the reconstructed image by means of histogram specification. LBP is capable of encoding texture and shape information of the preprocessed images. The histogram graphs built from multi-scale rotation invariant LBPs are combined to serve as feature for further recognition. Template matching is adopted to classify facial expressions for its simplicity. The experimental results show that the proposed approach has good performance in efficiency and accuracy.展开更多
Accurately recognizing facial expressions is essential for effective social interactions.Non-human primates(NHPs)are widely used in the study of the neural mechanisms underpinning facial expression processing,yet it r...Accurately recognizing facial expressions is essential for effective social interactions.Non-human primates(NHPs)are widely used in the study of the neural mechanisms underpinning facial expression processing,yet it remains unclear how well monkeys can recognize the facial expressions of other species such as humans.In this study,we systematically investigated how monkeys process the facial expressions of conspecifics and humans using eye-tracking technology and sophisticated behavioral tasks,namely the temporal discrimination task(TDT)and face scan task(FST).We found that monkeys showed prolonged subjective time perception in response to Negative facial expressions in monkeys while showing longer reaction time to Negative facial expressions in humans.Monkey faces also reliably induced divergent pupil contraction in response to different expressions,while human faces and scrambled monkey faces did not.Furthermore,viewing patterns in the FST indicated that monkeys only showed bias toward emotional expressions upon observing monkey faces.Finally,masking the eye region marginally decreased the viewing duration for monkey faces but not for human faces.By probing facial expression processing in monkeys,our study demonstrates that monkeys are more sensitive to the facial expressions of conspecifics than those of humans,thus shedding new light on inter-species communication through facial expressions between NHPs and humans.展开更多
For socialized animals,such as prinlates,emotions arc the expression of internal states,which may be recognized by others to adjust an in dividual's potential actions(Girard and Bellone 2020).Facial expressions ar...For socialized animals,such as prinlates,emotions arc the expression of internal states,which may be recognized by others to adjust an in dividual's potential actions(Girard and Bellone 2020).Facial expressions are therefore important signals in communication(e.g.,happy or in pain)and can help individuals understand potential meanings between each other(Dolensek et al.2020).Facial expressions can be expressed and processed freely and are useful in social interactions and bonding(Waller et al.2016).展开更多
The realization of natural and authentic facial expressions in humanoid robots poses a challenging and prominent research domain,encompassing interdisciplinary facets including mechanical design,sensing and actuation ...The realization of natural and authentic facial expressions in humanoid robots poses a challenging and prominent research domain,encompassing interdisciplinary facets including mechanical design,sensing and actuation control,psychology,cognitive science,flexible electronics,artificial intelligence(AI),etc.We have traced the recent developments of humanoid robot heads for facial expressions,discussed major challenges in embodied AI and flexible electronics for facial expression recognition and generation,and highlighted future trends in this field.Developing humanoid robot heads with natural and authentic facial expressions demands collaboration in interdisciplinary fields such as multi-modal sensing,emotional computing,and human-robot interactions(HRIs)to advance the emotional anthropomorphism of humanoid robots,bridging the gap between humanoid robots and human beings and enabling seamless HRIs.展开更多
Pain is a strong symptom of diseases. Being an involuntary unpleasant feeling, it can be considered a reliable indicator of health issues. Pain has always been expressed verbally, but in some cases, traditional patien...Pain is a strong symptom of diseases. Being an involuntary unpleasant feeling, it can be considered a reliable indicator of health issues. Pain has always been expressed verbally, but in some cases, traditional patient self-reporting is not efficient. On one side, there are patients who have neurological disorders and cannot express themselves accurately, as well as patients who suddenly lose consciousness due to an abrupt faintness. On another side, medical staff working in crowded hospitals need to focus on emergencies and would opt for the automation of the task of looking after hospitalized patients during their entire stay, in order to notice any pain-related emergency. These issues can be tackled with deep learning. Knowing that pain is generally followed by spontaneous facial behaviors, facial expressions can be used as a substitute to verbal reporting, to express pain. In this paper, a convolutional neural network (CNN) model was built and trained to detect pain through patients’ facial expressions, using the UNBC-McMaster Shoulder Pain dataset. First, faces were detected from images using the Haarcascade Frontal Face Detector provided by OpenCV, and preprocessed through gray scaling, histogram equalization, face detection, image cropping, mean filtering, and normalization. Next, preprocessed images were fed into a CNN model which was built based on a modified version of the VGG16 architecture. The model was finally evaluated and fine-tuned in a continuous way based on its accuracy, which reached 92.5%.展开更多
Herein,a three-stage support vector machine(SVM)for facial expression recognition is proposed.The first stage comprises 21 SVMs,which are all the binary combinations of seven expressions.If one expression is dominant,...Herein,a three-stage support vector machine(SVM)for facial expression recognition is proposed.The first stage comprises 21 SVMs,which are all the binary combinations of seven expressions.If one expression is dominant,then the first stage will suffice;if two are dominant,then the second stage is used;and,if three are dominant,the third stage is used.These multilevel stages help reduce the possibility of experiencing an error as much as possible.Different image preprocessing stages are used to ensure that the features attained from the face detected have a meaningful and proper contribution to the classification stage.Facial expressions are created as a result of muscle movements on the face.These subtle movements are detected by the histogram-oriented gradient feature,because it is sensitive to the shapes of objects.The features attained are then used to train the three-stage SVM.Two different validation methods were used:the leave-one-out and K-fold tests.Experimental results on three databases(Japanese Female Facial Expression,Extended Cohn-Kanade Dataset,and Radboud Faces Database)show that the proposed system is competitive and has better performance compared with other works.展开更多
基金supported by China Academy of Railway Sciences Corporation Limited(No.2021YJ127).
文摘Artificial intelligence,such as deep learning technology,has advanced the study of facial expression recognition since facial expression carries rich emotional information and is significant for many naturalistic situations.To pursue a high facial expression recognition accuracy,the network model of deep learning is generally designed to be very deep while the model’s real-time performance is typically constrained and limited.With MobileNetV3,a lightweight model with a good accuracy,a further study is conducted by adding a basic ResNet module to each of its existing modules and an SSH(Single Stage Headless Face Detector)context module to expand the model’s perceptual field.In this article,the enhanced model named Res-MobileNetV3,could alleviate the subpar of real-time performance and compress the size of large network models,which can process information at a rate of up to 33 frames per second.Although the improved model has been verified to be slightly inferior to the current state-of-the-art method in aspect of accuracy rate on the publically available face expression datasets,it can bring a good balance on accuracy,real-time performance,model size and model complexity in practical applications.
文摘The dynamics of student engagement and emotional states significantly influence learning outcomes.Positive emotions resulting from successful task completion stand in contrast to negative affective states that arise from learning struggles or failures.Effective transitions to engagement occur upon problem resolution,while unresolved issues lead to frustration and subsequent boredom.This study proposes a Convolutional Neural Networks(CNN)based approach utilizing the Multi⁃source Academic Affective Engagement Dataset(MAAED)to categorize facial expressions into boredom,confusion,frustration,and yawning.This method provides an efficient and objective way to assess student engagement by extracting features from facial images.Recognizing and addressing negative affective states,such as confusion and boredom,is fundamental in creating supportive learning environments.Through automated frame extraction and model comparison,this study demonstrates reduced loss values with improving accuracy,showcasing the effectiveness of this method in objectively evaluating student engagement.Monitoring facial engagement with CNN using the MAAED dataset is essential for gaining insights into human behaviour and improving educational experiences.
基金supported by the National Natural Science Foundation of China under Grant No.62276051the Natural Science Foundation of Sichuan Province under Grant No.2023NSFSC0640Medical Industry Information Integration Collaborative Innovation Project of Yangtze Delta Region Institute under Grant No.U0723002。
文摘The estimation of pain intensity is critical for medical diagnosis and treatment of patients.With the development of image monitoring technology and artificial intelligence,automatic pain assessment based on facial expression and behavioral analysis shows a potential value in clinical applications.This paper reports a framework of convolutional neural network with global and local attention mechanism(GLA-CNN)for the effective detection of pain intensity at four-level thresholds using facial expression images.GLA-CNN includes two modules,namely global attention network(GANet)and local attention network(LANet).LANet is responsible for extracting representative local patch features of faces,while GANet extracts whole facial features to compensate for the ignored correlative features between patches.In the end,the global correlational and local subtle features are fused for the final estimation of pain intensity.Experiments under the UNBC-McMaster Shoulder Pain database demonstrate that GLA-CNN outperforms other state-of-the-art methods.Additionally,a visualization analysis is conducted to present the feature map of GLA-CNN,intuitively showing that it can extract not only local pain features but also global correlative facial ones.Our study demonstrates that pain assessment based on facial expression is a non-invasive and feasible method,and can be employed as an auxiliary pain assessment tool in clinical practice.
基金The National Natural Science Foundation of China (No.60503023,60872160)the Natural Science Foundation for Universities ofJiangsu Province (No.08KJD520009)the Intramural Research Foundationof Nanjing University of Information Science and Technology(No.Y603)
文摘A novel fuzzy linear discriminant analysis method by the canonical correlation analysis （fuzzy-LDA/CCA）is presented and applied to the facial expression recognition. The fuzzy method is used to evaluate the degree of the class membership to which each training sample belongs. CCA is then used to establish the relationship between each facial image and the corresponding class membership vector, and the class membership vector of a test image is estimated using this relationship. Moreover, the fuzzy-LDA/CCA method is also generalized to deal with nonlinear discriminant analysis problems via kernel method. The performance of the proposed method is demonstrated using real data.
文摘It is unknown if the ability of Portuguese in the identification of NimStim data set,which was created in America to provide facial expressions that could be recognized by untrained people,is(or not)similar to the Americans.To test this hypothesis the performance of Portuguese in the recognition of Happiness,Surprise,Sadness,Fear,Disgust and Anger NimStim facial expressions was compared with the Americans,but no significant differences were found.In both populations the easiest emotion to identify was Happiness while Fear was the most difficult one.However,with exception for Surprise,Portuguese tend to show a lower accuracy rate for all the emotions studied.Results highlighted some cultural differences.
基金supported by the Academy of Finland(267581)the D2I SHOK Project from Digile Oy as well as Nokia Technologies(Tampere,Finland)
文摘Facial expression recognition is a hot topic in computer vision, but it remains challenging due to the feature inconsistency caused by person-specific 'characteristics of facial expressions. To address such a challenge, and inspired by the recent success of deep identity network (DeepID-Net) for face identification, this paper proposes a novel deep learning based framework for recognising human expressions with facial images. Compared to the existing deep learning methods, our proposed framework, which is based on multi-scale global images and local facial patches, can significantly achieve a better performance on facial expression recognition. Finally, we verify the effectiveness of our proposed framework through experiments on the public benchmarking datasets JAFFE and extended Cohn-Kanade (CK+).
基金supported by National Natural Science Foundation of China(No.61273339)
文摘In expression recognition, feature representation is critical for successful recognition since it contains distinctive information of expressions. In this paper, a new approach for representing facial expression features is proposed with its objective to describe features in an effective and efficient way in order to improve the recognition performance. The method combines the facial action coding system(FACS) and 'uniform' local binary patterns(LBP) to represent facial expression features from coarse to fine. The facial feature regions are extracted by active shape models(ASM) based on FACS to obtain the gray-level texture. Then, LBP is used to represent expression features for enhancing the discriminant. A facial expression recognition system is developed based on this feature extraction method by using K nearest neighborhood(K-NN) classifier to recognize facial expressions. Finally, experiments are carried out to evaluate this feature extraction method. The significance of removing the unrelated facial regions and enhancing the discrimination ability of expression features in the recognition process is indicated by the results, in addition to its convenience.
基金supported by the National Natural Science Foundation of China(61403422,61273102)the Hubei Provincial Natural Science Foundation of China(2015CFA010)+1 种基金the Ⅲ Project(B17040)the Fundamental Research Funds for National University,China University of Geosciences(Wuhan)
文摘A facial expression emotion recognition based human-robot interaction(FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on2D-Gabor, uniform local binary pattern(LBP) operator, and multiclass extreme learning machine(ELM) classifier is presented,which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios,i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.
基金supported by research grants from the National Natural Science Foundation of China (No. 81071099)the Liaoning Science and Technology Foundation (No. 2008225010-14)Doctoral Foundation of the First Affiliated Hospital in China Medical University (No. 2010)
文摘Functional magnetic resonance imaging was used during emotion recognition to identify changes in functional brain activation in 21 first-episode, treatment-naive major depressive disorder patients before and after antidepressant treatment. Following escitalopram oxalate treatment, patients exhibited decreased activation in bilateral precentral gyrus, bilateral middle frontal gyrus, left middle temporal gyrus, bilateral postcentral gyrus, left cingulate and right parahippocampal gyrus, and increased activation in right superior frontal gyrus, bilateral superior parietal Iobule and left occipital gyrus during sad facial expression recognition. After antidepressant treatment, patients also exhibited decreased activation in the bilateral middle frontal gyrus, bilateral cingulate and right parahippocampal gyrus, and increased activation in the right inferior frontal gyrus, left fusiform gyrus and right precuneus during happy facial expression recognition. Our experimental findings indicate that the limbic-cortical network might be a key target region for antidepressant treatment in major depressive disorder.
文摘Facial Expression Recognition(FER)has been an interesting area of research in places where there is human-computer interaction.Human psychol-ogy,emotions and behaviors can be analyzed in FER.Classifiers used in FER have been perfect on normal faces but have been found to be constrained in occluded faces.Recently,Deep Learning Techniques(DLT)have gained popular-ity in applications of real-world problems including recognition of human emo-tions.The human face reflects emotional states and human intentions.An expression is the most natural and powerful way of communicating non-verbally.Systems which form communications between the two are termed Human Machine Interaction(HMI)systems.FER can improve HMI systems as human expressions convey useful information to an observer.This paper proposes a FER scheme called EECNN(Enhanced Convolution Neural Network with Atten-tion mechanism)to recognize seven types of human emotions with satisfying results in its experiments.Proposed EECNN achieved 89.8%accuracy in classi-fying the images.
基金funded by Anhui Province Quality Engineering Project No.2021jyxm0801Natural Science Foundation of Anhui University of Chinese Medicine under Grant Nos.2020zrzd18,2019zrzd11+1 种基金Humanity Social Science foundation Grants 2021rwzd20,2020rwzd07Anhui University of Chinese Medicine Quality Engineering Projects No.2021zlgc046.
文摘For the problems of complex model structure and too many training parameters in facial expression recognition algorithms,we proposed a residual network structure with a multi-headed channel attention(MCA)module.The migration learning algorithm is used to pre-train the convolutional layer parameters and mitigate the overfitting caused by the insufficient number of training samples.The designed MCA module is integrated into the ResNet18 backbone network.The attention mechanism highlights important information and suppresses irrelevant information by assigning different coefficients or weights,and the multi-head structure focuses more on the local features of the pictures,which improves the efficiency of facial expression recognition.Experimental results demonstrate that the model proposed in this paper achieves excellent recognition results in Fer2013,CK+and Jaffe datasets,with accuracy rates of 72.7%,98.8%and 93.33%,respectively.
文摘Local binary pattern(LBP)is an important method for texture feature extraction of facial expression.However,it also has the shortcomings of high dimension,slow feature extraction and noeffective local or global features extracted.To solve these problems,a facial expression feature extraction method is proposed based on improved LBP.Firstly,LBP is converted into double local binary pattern(DLBP).Then by combining Taylor expansion(TE)with DLBP,DLBP-TE algorithm is obtained.Finally,the DLBP-TE algorithm combined with extreme learning machine(ELM)is applied in seven kinds of ficial expression images and the corresponding experiments are carried out in Japanese adult female facial expression(JAFFE)database.The results show that the proposed method can significantly improve facial expression recognition rate.
文摘In computer vision,emotion recognition using facial expression images is considered an important research issue.Deep learning advances in recent years have aided in attaining improved results in this issue.According to recent studies,multiple facial expressions may be included in facial photographs representing a particular type of emotion.It is feasible and useful to convert face photos into collections of visual words and carry out global expression recognition.The main contribution of this paper is to propose a facial expression recognitionmodel(FERM)depending on an optimized Support Vector Machine(SVM).To test the performance of the proposed model(FERM),AffectNet is used.AffectNet uses 1250 emotion-related keywords in six different languages to search three major search engines and get over 1,000,000 facial photos online.The FERM is composed of three main phases:(i)the Data preparation phase,(ii)Applying grid search for optimization,and(iii)the categorization phase.Linear discriminant analysis(LDA)is used to categorize the data into eight labels(neutral,happy,sad,surprised,fear,disgust,angry,and contempt).Due to using LDA,the performance of categorization via SVM has been obviously enhanced.Grid search is used to find the optimal values for hyperparameters of SVM(C and gamma).The proposed optimized SVM algorithm has achieved an accuracy of 99%and a 98%F1 score.
基金supported by the National Natural Science Foundation of China,No.81473769(to WW),81772430(to WW)a grant from the Training Program of Innovation and Entrepreneurship for Undergraduates of Southern Medical University of Guangdong Province of China in 2016,No.201612121057(to WW)
文摘OBJECTIVE: The objective of this study is to summarize and analyze the brain signal patterns of empathy for pain caused by facial expressions of pain utilizing activation likelihood estimation, a meta-analysis method. DATA SOURCES: Studies concerning the brain mechanism were searched from the Science Citation Index, Science Direct, PubMed, DeepDyve, Cochrane Library, SinoMed, Wanfang, VIP, China National Knowledge Infrastructure, and other databases, such as SpringerLink, AMA, Science Online, Wiley Online, were collected. A time limitation of up to 13 December 2016 was applied to this study. DATA SELECTION: Studies presenting with all of the following criteria were considered for study inclusion: Use of functional magnetic resonance imaging, neutral and pained facial expression stimuli, involvement of adult healthy human participants over 18 years of age, whose empathy ability showed no difference from the healthy adult, a painless basic state, results presented in Talairach or Montreal Neurological Institute coordinates, multiple studies by the same team as long as they used different raw data. OUTCOME MEASURES: Activation likelihood estimation was used to calculate the combined main activated brain regions under the stimulation of pained facial expression. RESULTS: Eight studies were included, containing 178 subjects. Meta-analysis results suggested that the anterior cingulate cortex(BA32), anterior central gyrus(BA44), fusiform gyrus, and insula(BA13) were activated positively as major brain areas under the stimulation of pained facial expression. CONCLUSION: Our study shows that pained facial expression alone, without viewing of painful stimuli, activated brain regions related to pain empathy, further contributing to revealing the brain's mechanisms of pain empathy.
文摘In this paper, a novel method based on dual-tree complex wavelet transform(DT-CWT) and rotation invariant local binary pattern(LBP) for facial expression recognition is proposed. The quarter sample shift (Q-shift) DT-CWT can provide a group delay of 1/4 of a sample period, and satisfy the usual 2-band filter bank constraints of no aliasing and perfect reconstruction. To resolve illumination variation in expression verification, low-frequency coefficients produced by DT-CWT are set zeroes, high-frequency coefficients are used for reconstructing the image, and basic LBP histogram is mapped on the reconstructed image by means of histogram specification. LBP is capable of encoding texture and shape information of the preprocessed images. The histogram graphs built from multi-scale rotation invariant LBPs are combined to serve as feature for further recognition. Template matching is adopted to classify facial expressions for its simplicity. The experimental results show that the proposed approach has good performance in efficiency and accuracy.
基金supported by the National Natural Science Foundation of China (U20A2017)Guangdong Basic and Applied Basic Research Foundation (2022A1515010134,2022A1515110598)+2 种基金Youth Innovation Promotion Association of Chinese Academy of Sciences (2017120)Shenzhen-Hong Kong Institute of Brain Science–Shenzhen Fundamental Research Institutions (NYKFKT2019009)Shenzhen Technological Research Center for Primate Translational Medicine (F-2021-Z99-504979)。
文摘Accurately recognizing facial expressions is essential for effective social interactions.Non-human primates(NHPs)are widely used in the study of the neural mechanisms underpinning facial expression processing,yet it remains unclear how well monkeys can recognize the facial expressions of other species such as humans.In this study,we systematically investigated how monkeys process the facial expressions of conspecifics and humans using eye-tracking technology and sophisticated behavioral tasks,namely the temporal discrimination task(TDT)and face scan task(FST).We found that monkeys showed prolonged subjective time perception in response to Negative facial expressions in monkeys while showing longer reaction time to Negative facial expressions in humans.Monkey faces also reliably induced divergent pupil contraction in response to different expressions,while human faces and scrambled monkey faces did not.Furthermore,viewing patterns in the FST indicated that monkeys only showed bias toward emotional expressions upon observing monkey faces.Finally,masking the eye region marginally decreased the viewing duration for monkey faces but not for human faces.By probing facial expression processing in monkeys,our study demonstrates that monkeys are more sensitive to the facial expressions of conspecifics than those of humans,thus shedding new light on inter-species communication through facial expressions between NHPs and humans.
基金This study was supported by the Key Program of the National Natural Science Fund(31730104)National Natural Science Foundation of China(31801981)+5 种基金Strategic Priority Research Program of the Chinese Academy of Sciences(XDB31020302)National Key Program of Research and Development,Ministry of Science and Technology(2016YFC0503200)Innovation Capability Support Program of Shaanxi(2020KJXX-008)Foundation of Shaanxi Academy of Sciences of China(2016K-20,2018K-16-04,2019ZY-JCTJ-06)Key Research and Development Program of Shaanxi Province of China(2018PT-04)One Institute One Brand Foundation of Shaanxi Academy of Sciences(2020k-01).
文摘For socialized animals,such as prinlates,emotions arc the expression of internal states,which may be recognized by others to adjust an in dividual's potential actions(Girard and Bellone 2020).Facial expressions are therefore important signals in communication(e.g.,happy or in pain)and can help individuals understand potential meanings between each other(Dolensek et al.2020).Facial expressions can be expressed and processed freely and are useful in social interactions and bonding(Waller et al.2016).
基金supported by the National Natural Science Foundation of China(nos.52188102 and 51925503)the Science and Technology Development Fund of Macao SAR(file na.0117/2024/AMJ)+1 种基金Zhuhai UM Science&Technology Research Institute(CP-009-2024)the State Key Laboratory of Intelligent Manufacturing Equipment and Tech-nology(IMETKF2024003),HUST,Wuhan,China.
文摘The realization of natural and authentic facial expressions in humanoid robots poses a challenging and prominent research domain,encompassing interdisciplinary facets including mechanical design,sensing and actuation control,psychology,cognitive science,flexible electronics,artificial intelligence(AI),etc.We have traced the recent developments of humanoid robot heads for facial expressions,discussed major challenges in embodied AI and flexible electronics for facial expression recognition and generation,and highlighted future trends in this field.Developing humanoid robot heads with natural and authentic facial expressions demands collaboration in interdisciplinary fields such as multi-modal sensing,emotional computing,and human-robot interactions(HRIs)to advance the emotional anthropomorphism of humanoid robots,bridging the gap between humanoid robots and human beings and enabling seamless HRIs.
文摘Pain is a strong symptom of diseases. Being an involuntary unpleasant feeling, it can be considered a reliable indicator of health issues. Pain has always been expressed verbally, but in some cases, traditional patient self-reporting is not efficient. On one side, there are patients who have neurological disorders and cannot express themselves accurately, as well as patients who suddenly lose consciousness due to an abrupt faintness. On another side, medical staff working in crowded hospitals need to focus on emergencies and would opt for the automation of the task of looking after hospitalized patients during their entire stay, in order to notice any pain-related emergency. These issues can be tackled with deep learning. Knowing that pain is generally followed by spontaneous facial behaviors, facial expressions can be used as a substitute to verbal reporting, to express pain. In this paper, a convolutional neural network (CNN) model was built and trained to detect pain through patients’ facial expressions, using the UNBC-McMaster Shoulder Pain dataset. First, faces were detected from images using the Haarcascade Frontal Face Detector provided by OpenCV, and preprocessed through gray scaling, histogram equalization, face detection, image cropping, mean filtering, and normalization. Next, preprocessed images were fed into a CNN model which was built based on a modified version of the VGG16 architecture. The model was finally evaluated and fine-tuned in a continuous way based on its accuracy, which reached 92.5%.
文摘Herein,a three-stage support vector machine(SVM)for facial expression recognition is proposed.The first stage comprises 21 SVMs,which are all the binary combinations of seven expressions.If one expression is dominant,then the first stage will suffice;if two are dominant,then the second stage is used;and,if three are dominant,the third stage is used.These multilevel stages help reduce the possibility of experiencing an error as much as possible.Different image preprocessing stages are used to ensure that the features attained from the face detected have a meaningful and proper contribution to the classification stage.Facial expressions are created as a result of muscle movements on the face.These subtle movements are detected by the histogram-oriented gradient feature,because it is sensitive to the shapes of objects.The features attained are then used to train the three-stage SVM.Two different validation methods were used:the leave-one-out and K-fold tests.Experimental results on three databases(Japanese Female Facial Expression,Extended Cohn-Kanade Dataset,and Radboud Faces Database)show that the proposed system is competitive and has better performance compared with other works.
