This paper proposes a method to recognize human-object interactions by modeling context between human actions and interacted objects.Human-object interaction recognition is a challenging task due to severe occlusion b...This paper proposes a method to recognize human-object interactions by modeling context between human actions and interacted objects.Human-object interaction recognition is a challenging task due to severe occlusion between human and objects during the interacting process.Since that human actions and interacted objects provide strong context information,i.e.some actions are usually related to some specific objects,the accuracy of recognition is significantly improved for both of them.Through the proposed method,both global and local temporal features from skeleton sequences are extracted to model human actions.In the meantime,kernel features are utilized to describe interacted objects.Finally,all possible solutions from actions and objects are optimized by modeling the context between them.The results of experiments demonstrate the effectiveness of our method.展开更多
Video emotion recognition is widely used due to its alignment with the temporal characteristics of human emotional expression,but existingmodels have significant shortcomings.On the one hand,Transformermultihead self-...Video emotion recognition is widely used due to its alignment with the temporal characteristics of human emotional expression,but existingmodels have significant shortcomings.On the one hand,Transformermultihead self-attention modeling of global temporal dependency has problems of high computational overhead and feature similarity.On the other hand,fixed-size convolution kernels are often used,which have weak perception ability for emotional regions of different scales.Therefore,this paper proposes a video emotion recognition model that combines multi-scale region-aware convolution with temporal interactive sampling.In terms of space,multi-branch large-kernel stripe convolution is used to perceive emotional region features at different scales,and attention weights are generated for each scale feature.In terms of time,multi-layer odd-even down-sampling is performed on the time series,and oddeven sub-sequence interaction is performed to solve the problem of feature similarity,while reducing computational costs due to the linear relationship between sampling and convolution overhead.This paper was tested on CMU-MOSI,CMU-MOSEI,and Hume Reaction.The Acc-2 reached 83.4%,85.2%,and 81.2%,respectively.The experimental results show that the model can significantly improve the accuracy of emotion recognition.展开更多
Recognizing human interactions in RGB videos is a critical task in computer vision,with applications in video surveillance.Existing deep learning-based architectures have achieved strong results,but are computationall...Recognizing human interactions in RGB videos is a critical task in computer vision,with applications in video surveillance.Existing deep learning-based architectures have achieved strong results,but are computationally intensive,sensitive to video resolution changes and often fail in crowded scenes.We propose a novel hybrid system that is computationally efficient,robust to degraded video quality and able to filter out irrelevant individuals,making it suitable for real-life use.The system leverages multi-modal handcrafted features for interaction representation and a deep learning classifier for capturing complex dependencies.Using Mask R-CNN and YOLO11-Pose,we extract grayscale silhouettes and keypoint coordinates of interacting individuals,while filtering out irrelevant individuals using a proposed algorithm.From these,we extract silhouette-based features(local ternary pattern and histogram of optical flow)and keypoint-based features(distances,angles and velocities)that capture distinct spatial and temporal information.A Bidirectional Long Short-Term Memory network(BiLSTM)then classifies the interactions.Extensive experiments on the UT Interaction,SBU Kinect Interaction and the ISR-UOL 3D social activity datasets demonstrate that our system achieves competitive accuracy.They also validate the effectiveness of the chosen features and classifier,along with the proposed system’s computational efficiency and robustness to occlusion.展开更多
Lip language provides a silent,intuitive,and efficient mode of communication,offering a promising solution for individuals with speech impairments.Its articulation relies on complex movements of the jaw and the muscle...Lip language provides a silent,intuitive,and efficient mode of communication,offering a promising solution for individuals with speech impairments.Its articulation relies on complex movements of the jaw and the muscles surrounding it.However,the accurate and real-time acquisition and decoding of these movements into reliable silent speech signals remains a significant challenge.In this work,we propose a real-time silent speech recognition system,which integrates a triboelectric nanogenerator-based flexible pressure sensor(FPS)with a deep learning framework.The FPS employs a porous pyramid-structured silicone film as the negative triboelectric layer,enabling highly sensitive pressure detection in the low-force regime(1 V N^(-1) for 0-10 N and 4.6 V N^(-1) for 10-24 N).This allows it to precisely capture jaw movements during speech and convert them into electrical signals.To decode the signals,we proposed a convolutional neural networklong short-term memory(CNN-LSTM)hybrid network,combining CNN and LSTM model to extract both local spatial features and temporal dynamics.The model achieved 95.83%classification accuracy in 30 categories of daily words.Furthermore,the decoded silent speech signals can be directly translated into executable commands for contactless and precise control of the smartphone.The system can also be connected to AR glasses,offering a novel human-machine interaction approach with promising potential in AR/VR applications.展开更多
Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions...Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions.Existing methods can be categorized into motion-level,event-level,and story-level ones based on spatiotemporal granularity.However,single-modal approaches struggle to capture complex behavioral semantics and human factors.Therefore,in recent years,vision-language models(VLMs)have been introduced into this field,providing new research perspectives for VAR.In this paper,we systematically review spatiotemporal hierarchical methods in VAR and explore how the introduction of large models has advanced the field.Additionally,we propose the concept of“Factor”to identify and integrate key information from both visual and textual modalities,enhancing multimodal alignment.We also summarize various multimodal alignment methods and provide in-depth analysis and insights into future research directions.展开更多
With the growing application of intelligent robots in service,manufacturing,and medical fields,efficient and natural interaction between humans and robots has become key to improving collaboration efficiency and user ...With the growing application of intelligent robots in service,manufacturing,and medical fields,efficient and natural interaction between humans and robots has become key to improving collaboration efficiency and user experience.Gesture recognition,as an intuitive and contactless interaction method,can overcome the limitations of traditional interfaces and enable real-time control and feedback of robot movements and behaviors.This study first reviews mainstream gesture recognition algorithms and their application on different sensing platforms(RGB cameras,depth cameras,and inertial measurement units).It then proposes a gesture recognition method based on multimodal feature fusion and a lightweight deep neural network that balances recognition accuracy with computational efficiency.At system level,a modular human-robot interaction architecture is constructed,comprising perception,decision,and execution layers,and gesture commands are transmitted and mapped to robot actions in real time via the ROS communication protocol.Through multiple comparative experiments on public gesture datasets and a self-collected dataset,the proposed method’s superiority is validated in terms of accuracy,response latency,and system robustness,while user-experience tests assess the interface’s usability.The results provide a reliable technical foundation for robot collaboration and service in complex scenarios,offering broad prospects for practical application and deployment.展开更多
This study presents a hybrid CNN-Transformer model for real-time recognition of affective tactile biosignals.The proposed framework combines convolutional neural networks(CNNs)to extract spatial and local temporal fea...This study presents a hybrid CNN-Transformer model for real-time recognition of affective tactile biosignals.The proposed framework combines convolutional neural networks(CNNs)to extract spatial and local temporal features with the Transformer encoder that captures long-range dependencies in time-series data through multi-head attention.Model performance was evaluated on two widely used tactile biosignal datasets,HAART and CoST,which contain diverse affective touch gestures recorded from pressure sensor arrays.TheCNN-Transformer model achieved recognition rates of 93.33%on HAART and 80.89%on CoST,outperforming existing methods on both benchmarks.By incorporating temporal windowing,the model enables instantaneous prediction,improving generalization across gestures of varying duration.These results highlight the effectiveness of deep learning for tactile biosignal processing and demonstrate the potential of theCNN-Transformer approach for future applications in wearable sensors,affective computing,and biomedical monitoring.展开更多
A facial expression emotion recognition based human-robot interaction(FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize huma...A facial expression emotion recognition based human-robot interaction(FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on2D-Gabor, uniform local binary pattern(LBP) operator, and multiclass extreme learning machine(ELM) classifier is presented,which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios,i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.展开更多
Molecularly imprinted polymer, exhibiting considerable enantioselectivity for L-mandelic acid, was prepared using metal coordination-chelation interaction. By evaluating the recognition characteristics in the chromato...Molecularly imprinted polymer, exhibiting considerable enantioselectivity for L-mandelic acid, was prepared using metal coordination-chelation interaction. By evaluating the recognition characteristics in the chromatographic mode, the recognition interactions were proposed: specific and nonspecific metal coordination-chelation interaction and hydrophobic interaction were responsible for substrate binding on metal-complexing imprinted polymer; while the selective recognition only came from specific metal coordination-chelation interaction and specific hydrophobic interaction.展开更多
The recognition interaction of Rhodamine B(RB) with DNA was studied in a Britton-Robinson (B-R) buffer solution with pH=7.5 at a glassy carbon electrode by electrochemical techniques. RB shows an irreversible oxidatio...The recognition interaction of Rhodamine B(RB) with DNA was studied in a Britton-Robinson (B-R) buffer solution with pH=7.5 at a glassy carbon electrode by electrochemical techniques. RB shows an irreversible oxidation peak at +0.92 V(vs. SCE). After the addition of DNA in the RB solution, the peak current of RB decreased apparently without the shift of the peak potential. The electrochemical parameters such as the charge transfer coefficient α and the electrode reaction rate constant k s of the interaction system were carefully studied. The parameters did not change before and after the addition of DNA, which indicated that an electrochemical non-active complex had been formed, so the concentration of RB in the solution decreased and the peak current decreased correspondingly. The binding ratio of RB to DNA was 2∶1 with a binding constant of 2.66×10 9.展开更多
A new method for interaction recognition based on sparse representation of feature covariance matrices was presented.Firstly,the dense trajectories(DT)extracted from the video were clustered into different groups to e...A new method for interaction recognition based on sparse representation of feature covariance matrices was presented.Firstly,the dense trajectories(DT)extracted from the video were clustered into different groups to eliminate the irrelevant trajectories,which could greatly reduce the noise influence on feature extraction.Then,the trajectory tunnels were characterized by means of feature covariance matrices.In this way,the discriminative descriptors could be extracted,which was also an effective solution to the problem that the description of the feature second-order statistics is insufficient.After that,an over-complete dictionary was learned with the descriptors and all the descriptors were encoded using sparse coding(SC).Classification was achieved using multiple instance learning(MIL),which was more suitable for complex environments.The proposed method was tested and evaluated on the WEB Interaction dataset and the UT interaction dataset.The experimental results demonstrated the superior efficiency.展开更多
The recognition interaction of rhodamine B (RB) with DNA was studied in pH 7.5 Britton-Robinson (B-R) buffer solution by electrochemical techniques. An irreversible oxidation peak at glassy carbon electrode was obtain...The recognition interaction of rhodamine B (RB) with DNA was studied in pH 7.5 Britton-Robinson (B-R) buffer solution by electrochemical techniques. An irreversible oxidation peak at glassy carbon electrode was obtained at +0.92V (vs. SCE). After the addition of DNA into the RB solution, the peak current of RB decreased apparently without the shift of peak potential. The electrochemical parameters such as the charge transfer coefficient a and the electrode reaction standard rate constant ks of RB in the absence and presence of DNA were determined, which did not change, indicating that a non-electroactive complex was formed, so the concentration of RB in the solution decreased and the peak current decreased correspondingly.展开更多
Recently,vision-based gesture recognition(VGR)has become a hot research spot in human-computer interaction(HCI).Unlike other gesture recognition methods with data gloves or other wearable sensors,vision-based gesture ...Recently,vision-based gesture recognition(VGR)has become a hot research spot in human-computer interaction(HCI).Unlike other gesture recognition methods with data gloves or other wearable sensors,vision-based gesture recognition could lead to more natural and intuitive HCI interactions.This paper reviews the state-of-the-art vision-based gestures recognition methods,from different stages of gesture recognition process,i.e.,(1)image acquisition and pre-processing,(2)gesture segmentation,(3)gesture tracking,(4)feature extraction,and(5)gesture classification.This paper also analyzes the advantages and disadvantages of these various methods in detail.Finally,the challenges of vision-based gesture recognition in haptic rendering and future research directions are discussed.展开更多
Thermoresponsive biotinylated dendronized copolymers carrying dendritic oligoethylene glycol（OEG）pendants were prepared via free radical polymerization,and their protein recognitions based on biotin-avidin interacti...Thermoresponsive biotinylated dendronized copolymers carrying dendritic oligoethylene glycol（OEG）pendants were prepared via free radical polymerization,and their protein recognitions based on biotin-avidin interaction investigated.Both first（PG1） and second generation（PG2） dendronized copolymers were designed to examine possible thickness effects on the interaction between biotin and avidin.Inherited from the outstanding thermoresponsive properties from OEG dendrons,these biotinylated cylindrical copolymers show characteristic thermoresponsive behavior which provides an envelope to capture avidin through switching temperatures above or below their phase transition temperatures（T_（cp）s）.Thus,the recognition of polymer-supported biotin with avidin was investigated with UV/vis spectroscopy and dynamic laser light scattering.In contrast to the case for PG1,the increased thickness for copolymer PG2 hinders partially and inhibits the recognition of biotin moieties with avidin either below or above its T_（cp）.This demonstrates the significant architecture effects from dendronized polymers on the biotin moieties to shift onto periphery of the collapsed aggregates,which should be a prerequisite for protein recognition.These kinds of novel thermoresponsive copolymers may pave a way for the interesting biological applications in areas such as reversible activity control of enzyme or proteins,and for controlled delivery of drugs or genes.展开更多
Understanding the regulatory mechanism of self-assembly processes is a necessity to modulate nanostructures and their properties. Herein, we have studied the mechanism of self-assembly in the C3 symmetric 1,3,5-benzen...Understanding the regulatory mechanism of self-assembly processes is a necessity to modulate nanostructures and their properties. Herein, we have studied the mechanism of self-assembly in the C3 symmetric 1,3,5-benzentricarboxylic amino acid methyl ester enantiomers(TPE) in a mixed solvent system consisting of methanol and water. The resultant chiral structure was used for chiral recognition. The formation of chiral structures from the synergistic effect of multiple noncovalent interaction forces was confirmed by various techniques. Molecular dynamics simulations were used to characterize the time evolution of TPE structure and properties in solution. The theoretical results were consistent with the experimental results. Furthermore, the chiral structure assembled by the building blocks of TPE molecules was highly stereoselective for diamine compounds.展开更多
In the actual complex environment,the recognition accuracy of crop leaf disease is often not high.Inspired by the brain parallel interaction mechanism,a two-stream parallel interactive convolutional neural network(TSP...In the actual complex environment,the recognition accuracy of crop leaf disease is often not high.Inspired by the brain parallel interaction mechanism,a two-stream parallel interactive convolutional neural network(TSPI-CNN)is proposed to improve the recognition accuracy.TSPI-CNN includes a two-stream parallel network(TSP-Net)and a parallel interactive network(PI-Net).TSP-Net simulates the ventral and dorsal stream.PI-Net simulates the interaction between two pathways in the process of human brain visual information transmission.A large number of experiments shows that the proposed TSPI-CNN performs well on MK-D2,PlantVillage,Apple-3 leaf,and Cassava leaf datasets.Furthermore,the effect of numbers of interactions on the recognition performance of TSPI-CNN is discussed.The experimental results show that as the number of interactions increases,the recognition accuracy of the network also increases.Finally,the network is visualized to show the working mechanism of the network and provide enlightenment for future research.展开更多
Human Interaction Recognition(HIR)was one of the challenging issues in computer vision research due to the involvement of multiple individuals and their mutual interactions within video frames generated from their mov...Human Interaction Recognition(HIR)was one of the challenging issues in computer vision research due to the involvement of multiple individuals and their mutual interactions within video frames generated from their movements.HIR requires more sophisticated analysis than Human Action Recognition(HAR)since HAR focuses solely on individual activities like walking or running,while HIR involves the interactions between people.This research aims to develop a robust system for recognizing five common human interactions,such as hugging,kicking,pushing,pointing,and no interaction,from video sequences using multiple cameras.In this study,a hybrid Deep Learning(DL)and Machine Learning(ML)model was employed to improve classification accuracy and generalizability.The dataset was collected in an indoor environment with four-channel cameras capturing the five types of interactions among 13 participants.The data was processed using a DL model with a fine-tuned ResNet(Residual Networks)architecture based on 2D Convolutional Neural Network(CNN)layers for feature extraction.Subsequently,machine learning models were trained and utilized for interaction classification using six commonly used ML algorithms,including SVM,KNN,RF,DT,NB,and XGBoost.The results demonstrate a high accuracy of 95.45%in classifying human interactions.The hybrid approach enabled effective learning,resulting in highly accurate performance across different interaction types.Future work will explore more complex scenarios involving multiple individuals based on the application of this architecture.展开更多
Human-object interaction(HOIs)detection is a new branch of visual relationship detection,which plays an important role in the field of image understanding.Because of the complexity and diversity of image content,the d...Human-object interaction(HOIs)detection is a new branch of visual relationship detection,which plays an important role in the field of image understanding.Because of the complexity and diversity of image content,the detection of HOIs is still an onerous challenge.Unlike most of the current works for HOIs detection which only rely on the pairwise information of a human and an object,we propose a graph-based HOIs detection method that models context and global structure information.Firstly,to better utilize the relations between humans and objects,the detected humans and objects are regarded as nodes to construct a fully connected undirected graph,and the graph is pruned to obtain an HOI graph that only preserving the edges connecting human and object nodes.Then,in order to obtain more robust features of human and object nodes,two different attention-based feature extraction networks are proposed,which model global and local contexts respectively.Finally,the graph attention network is introduced to pass messages between different nodes in the HOI graph iteratively,and detect the potential HOIs.Experiments on V-COCO and HICO-DET datasets verify the effectiveness of the proposed method,and show that it is superior to many existing methods.展开更多
This article describes a pilot study aiming at generating social interactions between a humanoid robot and adolescents with autism spectrum disorder (ASD), through the practice of a gesture imitation game. The partici...This article describes a pilot study aiming at generating social interactions between a humanoid robot and adolescents with autism spectrum disorder (ASD), through the practice of a gesture imitation game. The participants were a 17-year-old young lady with ASD and intellectual deficit, and a control participant: a preadolescent with ASD but no intellectual deficit (Asperger syndrome). The game is comprised of four phases: greetings, pairing, imitation, and closing. Field educators were involved, playing specific roles: visual or physical inciter. The use of a robot allows for catching the participants’ attention, playing the imitation game for a longer period of time than with a human partner, and preventing the game partner’s negative facial expressions resulting from tiredness, impatience, or boredom. The participants’ behavior was observed in terms of initial approach towards the robot, positioning relative to the robot in terms of distance and orientation, reactions to the robot’s voice or moves, signs of happiness, and imitation attempts. Results suggest a more and more natural approach towards the robot during the sessions, as well as a higher level of social interaction, based on the variations of the parameters listed above. We use these preliminary results to draw the next steps of our research work as well as identify further perspectives, with this aim in mind: improving social interactions with adolescents with ASD and intellectual deficit, allowing for better integration of these people into our societies.展开更多
Molecular recognition of bioreceptors and enzymes relies on orthogonal interactions with small molecules within their cavity. To date, Chinese scientists have developed three types of strategies for introducing active...Molecular recognition of bioreceptors and enzymes relies on orthogonal interactions with small molecules within their cavity. To date, Chinese scientists have developed three types of strategies for introducing active sites inside the cavity of macrocyclic arenes to better mimic molecular recognition of bioreceptors and enzymes.The editorial aims to enlighten scientists in this field when they develop novel macrocycles for molecular recognition, supramolecular assembly, and applications.展开更多
文摘This paper proposes a method to recognize human-object interactions by modeling context between human actions and interacted objects.Human-object interaction recognition is a challenging task due to severe occlusion between human and objects during the interacting process.Since that human actions and interacted objects provide strong context information,i.e.some actions are usually related to some specific objects,the accuracy of recognition is significantly improved for both of them.Through the proposed method,both global and local temporal features from skeleton sequences are extracted to model human actions.In the meantime,kernel features are utilized to describe interacted objects.Finally,all possible solutions from actions and objects are optimized by modeling the context between them.The results of experiments demonstrate the effectiveness of our method.
基金supported,in part,by the National Nature Science Foundation of China under Grant 62272236,62376128in part,by the Natural Science Foundation of Jiangsu Province under Grant BK20201136,BK20191401.
文摘Video emotion recognition is widely used due to its alignment with the temporal characteristics of human emotional expression,but existingmodels have significant shortcomings.On the one hand,Transformermultihead self-attention modeling of global temporal dependency has problems of high computational overhead and feature similarity.On the other hand,fixed-size convolution kernels are often used,which have weak perception ability for emotional regions of different scales.Therefore,this paper proposes a video emotion recognition model that combines multi-scale region-aware convolution with temporal interactive sampling.In terms of space,multi-branch large-kernel stripe convolution is used to perceive emotional region features at different scales,and attention weights are generated for each scale feature.In terms of time,multi-layer odd-even down-sampling is performed on the time series,and oddeven sub-sequence interaction is performed to solve the problem of feature similarity,while reducing computational costs due to the linear relationship between sampling and convolution overhead.This paper was tested on CMU-MOSI,CMU-MOSEI,and Hume Reaction.The Acc-2 reached 83.4%,85.2%,and 81.2%,respectively.The experimental results show that the model can significantly improve the accuracy of emotion recognition.
基金supported and funded by Princess Nourah bint Abdulrahman University Researchers Supporting Project number(PNURSP2025R410),Princess Nourah bint Abdulrahman University,Riyadh,Saudi Arabia.
文摘Recognizing human interactions in RGB videos is a critical task in computer vision,with applications in video surveillance.Existing deep learning-based architectures have achieved strong results,but are computationally intensive,sensitive to video resolution changes and often fail in crowded scenes.We propose a novel hybrid system that is computationally efficient,robust to degraded video quality and able to filter out irrelevant individuals,making it suitable for real-life use.The system leverages multi-modal handcrafted features for interaction representation and a deep learning classifier for capturing complex dependencies.Using Mask R-CNN and YOLO11-Pose,we extract grayscale silhouettes and keypoint coordinates of interacting individuals,while filtering out irrelevant individuals using a proposed algorithm.From these,we extract silhouette-based features(local ternary pattern and histogram of optical flow)and keypoint-based features(distances,angles and velocities)that capture distinct spatial and temporal information.A Bidirectional Long Short-Term Memory network(BiLSTM)then classifies the interactions.Extensive experiments on the UT Interaction,SBU Kinect Interaction and the ISR-UOL 3D social activity datasets demonstrate that our system achieves competitive accuracy.They also validate the effectiveness of the chosen features and classifier,along with the proposed system’s computational efficiency and robustness to occlusion.
基金supported by the Natural Science Foundation of Fujian Province under Grant No.2024J010016Fujian Province Young and Middle aged Teacher Education Research Project No.JAT241317the Mindu Innovation Laboratory Project under Grant No.2020ZZ113.
文摘Lip language provides a silent,intuitive,and efficient mode of communication,offering a promising solution for individuals with speech impairments.Its articulation relies on complex movements of the jaw and the muscles surrounding it.However,the accurate and real-time acquisition and decoding of these movements into reliable silent speech signals remains a significant challenge.In this work,we propose a real-time silent speech recognition system,which integrates a triboelectric nanogenerator-based flexible pressure sensor(FPS)with a deep learning framework.The FPS employs a porous pyramid-structured silicone film as the negative triboelectric layer,enabling highly sensitive pressure detection in the low-force regime(1 V N^(-1) for 0-10 N and 4.6 V N^(-1) for 10-24 N).This allows it to precisely capture jaw movements during speech and convert them into electrical signals.To decode the signals,we proposed a convolutional neural networklong short-term memory(CNN-LSTM)hybrid network,combining CNN and LSTM model to extract both local spatial features and temporal dynamics.The model achieved 95.83%classification accuracy in 30 categories of daily words.Furthermore,the decoded silent speech signals can be directly translated into executable commands for contactless and precise control of the smartphone.The system can also be connected to AR glasses,offering a novel human-machine interaction approach with promising potential in AR/VR applications.
基金supported by the Zhejiang Provincial Natural Science Foundation of China(No.LQ23F030001)the National Natural Science Foundation of China(No.62406280)+5 种基金the Autism Research Special Fund of Zhejiang Foundation for Disabled Persons(No.2023008)the Liaoning Province Higher Education Innovative Talents Program Support Project(No.LR2019058)the Liaoning Province Joint Open Fund for Key Scientific and Technological Innovation Bases(No.2021-KF-12-05)the Central Guidance on Local Science and Technology Development Fund of Liaoning Province(No.2023JH6/100100066)the Key Laboratory for Biomedical Engineering of Ministry of Education,Zhejiang University,Chinain part by the Open Research Fund of the State Key Laboratory of Cognitive Neuroscience and Learning.
文摘Video action recognition(VAR)aims to analyze dynamic behaviors in videos and achieve semantic understanding.VAR faces challenges such as temporal dynamics,action-scene coupling,and the complexity of human interactions.Existing methods can be categorized into motion-level,event-level,and story-level ones based on spatiotemporal granularity.However,single-modal approaches struggle to capture complex behavioral semantics and human factors.Therefore,in recent years,vision-language models(VLMs)have been introduced into this field,providing new research perspectives for VAR.In this paper,we systematically review spatiotemporal hierarchical methods in VAR and explore how the introduction of large models has advanced the field.Additionally,we propose the concept of“Factor”to identify and integrate key information from both visual and textual modalities,enhancing multimodal alignment.We also summarize various multimodal alignment methods and provide in-depth analysis and insights into future research directions.
文摘With the growing application of intelligent robots in service,manufacturing,and medical fields,efficient and natural interaction between humans and robots has become key to improving collaboration efficiency and user experience.Gesture recognition,as an intuitive and contactless interaction method,can overcome the limitations of traditional interfaces and enable real-time control and feedback of robot movements and behaviors.This study first reviews mainstream gesture recognition algorithms and their application on different sensing platforms(RGB cameras,depth cameras,and inertial measurement units).It then proposes a gesture recognition method based on multimodal feature fusion and a lightweight deep neural network that balances recognition accuracy with computational efficiency.At system level,a modular human-robot interaction architecture is constructed,comprising perception,decision,and execution layers,and gesture commands are transmitted and mapped to robot actions in real time via the ROS communication protocol.Through multiple comparative experiments on public gesture datasets and a self-collected dataset,the proposed method’s superiority is validated in terms of accuracy,response latency,and system robustness,while user-experience tests assess the interface’s usability.The results provide a reliable technical foundation for robot collaboration and service in complex scenarios,offering broad prospects for practical application and deployment.
文摘This study presents a hybrid CNN-Transformer model for real-time recognition of affective tactile biosignals.The proposed framework combines convolutional neural networks(CNNs)to extract spatial and local temporal features with the Transformer encoder that captures long-range dependencies in time-series data through multi-head attention.Model performance was evaluated on two widely used tactile biosignal datasets,HAART and CoST,which contain diverse affective touch gestures recorded from pressure sensor arrays.TheCNN-Transformer model achieved recognition rates of 93.33%on HAART and 80.89%on CoST,outperforming existing methods on both benchmarks.By incorporating temporal windowing,the model enables instantaneous prediction,improving generalization across gestures of varying duration.These results highlight the effectiveness of deep learning for tactile biosignal processing and demonstrate the potential of theCNN-Transformer approach for future applications in wearable sensors,affective computing,and biomedical monitoring.
基金supported by the National Natural Science Foundation of China(61403422,61273102)the Hubei Provincial Natural Science Foundation of China(2015CFA010)+1 种基金the Ⅲ Project(B17040)the Fundamental Research Funds for National University,China University of Geosciences(Wuhan)
文摘A facial expression emotion recognition based human-robot interaction(FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on2D-Gabor, uniform local binary pattern(LBP) operator, and multiclass extreme learning machine(ELM) classifier is presented,which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios,i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.
文摘Molecularly imprinted polymer, exhibiting considerable enantioselectivity for L-mandelic acid, was prepared using metal coordination-chelation interaction. By evaluating the recognition characteristics in the chromatographic mode, the recognition interactions were proposed: specific and nonspecific metal coordination-chelation interaction and hydrophobic interaction were responsible for substrate binding on metal-complexing imprinted polymer; while the selective recognition only came from specific metal coordination-chelation interaction and specific hydrophobic interaction.
文摘The recognition interaction of Rhodamine B(RB) with DNA was studied in a Britton-Robinson (B-R) buffer solution with pH=7.5 at a glassy carbon electrode by electrochemical techniques. RB shows an irreversible oxidation peak at +0.92 V(vs. SCE). After the addition of DNA in the RB solution, the peak current of RB decreased apparently without the shift of the peak potential. The electrochemical parameters such as the charge transfer coefficient α and the electrode reaction rate constant k s of the interaction system were carefully studied. The parameters did not change before and after the addition of DNA, which indicated that an electrochemical non-active complex had been formed, so the concentration of RB in the solution decreased and the peak current decreased correspondingly. The binding ratio of RB to DNA was 2∶1 with a binding constant of 2.66×10 9.
基金Project(51678075) supported by the National Natural Science Foundation of ChinaProject(2017GK2271) supported by the Science and Technology Project of Hunan Province,China
文摘A new method for interaction recognition based on sparse representation of feature covariance matrices was presented.Firstly,the dense trajectories(DT)extracted from the video were clustered into different groups to eliminate the irrelevant trajectories,which could greatly reduce the noise influence on feature extraction.Then,the trajectory tunnels were characterized by means of feature covariance matrices.In this way,the discriminative descriptors could be extracted,which was also an effective solution to the problem that the description of the feature second-order statistics is insufficient.After that,an over-complete dictionary was learned with the descriptors and all the descriptors were encoded using sparse coding(SC).Classification was achieved using multiple instance learning(MIL),which was more suitable for complex environments.The proposed method was tested and evaluated on the WEB Interaction dataset and the UT interaction dataset.The experimental results demonstrated the superior efficiency.
基金The work was supported by the National Natural Science Foundation of China(Grant No 20375020).
文摘The recognition interaction of rhodamine B (RB) with DNA was studied in pH 7.5 Britton-Robinson (B-R) buffer solution by electrochemical techniques. An irreversible oxidation peak at glassy carbon electrode was obtained at +0.92V (vs. SCE). After the addition of DNA into the RB solution, the peak current of RB decreased apparently without the shift of peak potential. The electrochemical parameters such as the charge transfer coefficient a and the electrode reaction standard rate constant ks of RB in the absence and presence of DNA were determined, which did not change, indicating that a non-electroactive complex was formed, so the concentration of RB in the solution decreased and the peak current decreased correspondingly.
基金Supported by the National Natural Science Foundation of China(61773205,61773219)the Fundamental Research Funds for the Central Universities(NS2016032,NS2019018,Nanjing University of Aeronautics and Astronautics)+1 种基金the Scholarship from China Scholarship Council(201906835020)the Fundamental Research Funds for the Central Universities(the Graduate Student Innovation Base Open Fund Project of NUAA,kfjj20190307)。
文摘Recently,vision-based gesture recognition(VGR)has become a hot research spot in human-computer interaction(HCI).Unlike other gesture recognition methods with data gloves or other wearable sensors,vision-based gesture recognition could lead to more natural and intuitive HCI interactions.This paper reviews the state-of-the-art vision-based gestures recognition methods,from different stages of gesture recognition process,i.e.,(1)image acquisition and pre-processing,(2)gesture segmentation,(3)gesture tracking,(4)feature extraction,and(5)gesture classification.This paper also analyzes the advantages and disadvantages of these various methods in detail.Finally,the challenges of vision-based gesture recognition in haptic rendering and future research directions are discussed.
基金the National Natural Science Foundation of China(Nos.21374058,21474060 and 21574078)the Ph.D. Programs Foundation of Ministry of Education of China(No 201331081100166)the Shanghai Rising-Star Program(No.16QA1401800)
文摘Thermoresponsive biotinylated dendronized copolymers carrying dendritic oligoethylene glycol（OEG）pendants were prepared via free radical polymerization,and their protein recognitions based on biotin-avidin interaction investigated.Both first（PG1） and second generation（PG2） dendronized copolymers were designed to examine possible thickness effects on the interaction between biotin and avidin.Inherited from the outstanding thermoresponsive properties from OEG dendrons,these biotinylated cylindrical copolymers show characteristic thermoresponsive behavior which provides an envelope to capture avidin through switching temperatures above or below their phase transition temperatures（T_（cp）s）.Thus,the recognition of polymer-supported biotin with avidin was investigated with UV/vis spectroscopy and dynamic laser light scattering.In contrast to the case for PG1,the increased thickness for copolymer PG2 hinders partially and inhibits the recognition of biotin moieties with avidin either below or above its T_（cp）.This demonstrates the significant architecture effects from dendronized polymers on the biotin moieties to shift onto periphery of the collapsed aggregates,which should be a prerequisite for protein recognition.These kinds of novel thermoresponsive copolymers may pave a way for the interesting biological applications in areas such as reversible activity control of enzyme or proteins,and for controlled delivery of drugs or genes.
基金supported by the National Natural Science Foundation of China(No.21962003)the Natural Science Foundation of Jiangsu Province(No.BK20190056)the“Fundamental Research Funds for the Central Universities”(No.021514380014)。
文摘Understanding the regulatory mechanism of self-assembly processes is a necessity to modulate nanostructures and their properties. Herein, we have studied the mechanism of self-assembly in the C3 symmetric 1,3,5-benzentricarboxylic amino acid methyl ester enantiomers(TPE) in a mixed solvent system consisting of methanol and water. The resultant chiral structure was used for chiral recognition. The formation of chiral structures from the synergistic effect of multiple noncovalent interaction forces was confirmed by various techniques. Molecular dynamics simulations were used to characterize the time evolution of TPE structure and properties in solution. The theoretical results were consistent with the experimental results. Furthermore, the chiral structure assembled by the building blocks of TPE molecules was highly stereoselective for diamine compounds.
基金National Natural Science Foundation of China(Nos.61806051 and 61903078)Fundamental Research Funds for the Central Universities,China(Nos.2232021A-10 and 2232021D-32)Natural Science Foundation of Shanghai,China(No.20ZR1400400)。
文摘In the actual complex environment,the recognition accuracy of crop leaf disease is often not high.Inspired by the brain parallel interaction mechanism,a two-stream parallel interactive convolutional neural network(TSPI-CNN)is proposed to improve the recognition accuracy.TSPI-CNN includes a two-stream parallel network(TSP-Net)and a parallel interactive network(PI-Net).TSP-Net simulates the ventral and dorsal stream.PI-Net simulates the interaction between two pathways in the process of human brain visual information transmission.A large number of experiments shows that the proposed TSPI-CNN performs well on MK-D2,PlantVillage,Apple-3 leaf,and Cassava leaf datasets.Furthermore,the effect of numbers of interactions on the recognition performance of TSPI-CNN is discussed.The experimental results show that as the number of interactions increases,the recognition accuracy of the network also increases.Finally,the network is visualized to show the working mechanism of the network and provide enlightenment for future research.
基金supported by the National Research Foundation of Korea(NRF)grant funded by the Korea government(MSIT)(No.RS-2023-00218176)and the Soonchunhyang University Research Fund.
文摘Human Interaction Recognition(HIR)was one of the challenging issues in computer vision research due to the involvement of multiple individuals and their mutual interactions within video frames generated from their movements.HIR requires more sophisticated analysis than Human Action Recognition(HAR)since HAR focuses solely on individual activities like walking or running,while HIR involves the interactions between people.This research aims to develop a robust system for recognizing five common human interactions,such as hugging,kicking,pushing,pointing,and no interaction,from video sequences using multiple cameras.In this study,a hybrid Deep Learning(DL)and Machine Learning(ML)model was employed to improve classification accuracy and generalizability.The dataset was collected in an indoor environment with four-channel cameras capturing the five types of interactions among 13 participants.The data was processed using a DL model with a fine-tuned ResNet(Residual Networks)architecture based on 2D Convolutional Neural Network(CNN)layers for feature extraction.Subsequently,machine learning models were trained and utilized for interaction classification using six commonly used ML algorithms,including SVM,KNN,RF,DT,NB,and XGBoost.The results demonstrate a high accuracy of 95.45%in classifying human interactions.The hybrid approach enabled effective learning,resulting in highly accurate performance across different interaction types.Future work will explore more complex scenarios involving multiple individuals based on the application of this architecture.
基金Project(51678075)supported by the National Natural Science Foundation of ChinaProject(2017GK2271)supported by the Hunan Provincial Science and Technology Department,China。
文摘Human-object interaction(HOIs)detection is a new branch of visual relationship detection,which plays an important role in the field of image understanding.Because of the complexity and diversity of image content,the detection of HOIs is still an onerous challenge.Unlike most of the current works for HOIs detection which only rely on the pairwise information of a human and an object,we propose a graph-based HOIs detection method that models context and global structure information.Firstly,to better utilize the relations between humans and objects,the detected humans and objects are regarded as nodes to construct a fully connected undirected graph,and the graph is pruned to obtain an HOI graph that only preserving the edges connecting human and object nodes.Then,in order to obtain more robust features of human and object nodes,two different attention-based feature extraction networks are proposed,which model global and local contexts respectively.Finally,the graph attention network is introduced to pass messages between different nodes in the HOI graph iteratively,and detect the potential HOIs.Experiments on V-COCO and HICO-DET datasets verify the effectiveness of the proposed method,and show that it is superior to many existing methods.
文摘This article describes a pilot study aiming at generating social interactions between a humanoid robot and adolescents with autism spectrum disorder (ASD), through the practice of a gesture imitation game. The participants were a 17-year-old young lady with ASD and intellectual deficit, and a control participant: a preadolescent with ASD but no intellectual deficit (Asperger syndrome). The game is comprised of four phases: greetings, pairing, imitation, and closing. Field educators were involved, playing specific roles: visual or physical inciter. The use of a robot allows for catching the participants’ attention, playing the imitation game for a longer period of time than with a human partner, and preventing the game partner’s negative facial expressions resulting from tiredness, impatience, or boredom. The participants’ behavior was observed in terms of initial approach towards the robot, positioning relative to the robot in terms of distance and orientation, reactions to the robot’s voice or moves, signs of happiness, and imitation attempts. Results suggest a more and more natural approach towards the robot during the sessions, as well as a higher level of social interaction, based on the variations of the parameters listed above. We use these preliminary results to draw the next steps of our research work as well as identify further perspectives, with this aim in mind: improving social interactions with adolescents with ASD and intellectual deficit, allowing for better integration of these people into our societies.
文摘Molecular recognition of bioreceptors and enzymes relies on orthogonal interactions with small molecules within their cavity. To date, Chinese scientists have developed three types of strategies for introducing active sites inside the cavity of macrocyclic arenes to better mimic molecular recognition of bioreceptors and enzymes.The editorial aims to enlighten scientists in this field when they develop novel macrocycles for molecular recognition, supramolecular assembly, and applications.
