The increasing complexity of intelligent sensing environments,driven by the growth of Internet of Things technologies,has created a strong demand for neuromorphic systems capable of real-time,low-power multisensory pe...The increasing complexity of intelligent sensing environments,driven by the growth of Internet of Things technologies,has created a strong demand for neuromorphic systems capable of real-time,low-power multisensory perception.Traditional sensory architectures,constrained by single-modal processing and centralized computing,struggle to meet the requirements of diverse and dynamic input conditions.Multisensory neuromorphic devices offer a promising solution by mimicking the distributed,event-driven processing of biological systems.Recent efforts have explored synaptic devices and material systems that respond to various input modalities,including visual,tactile,thermal,and chemical stimuli.However,challenges remain in signal conversion,encoding compatibility,and the fusion of heterogeneous inputs without loss of unisensory information.This review provides a comprehensive overview of the physical mechanisms,device behaviors,and integration strategies that underpin signal processing in neuromorphic hardware.We highlight synaptic mechanisms conducive to cross-modal interaction,analyze representative signal fusion approaches at the device level,and discuss future directions for constructing efficient,scalable,and biologically inspired multisensory neuromorphic systems.展开更多
Understanding the neural underpinning of human gait and balance is one of the most pertinent challenges for 21st-century translational neuroscience due to the profound impact that falls and mobility disturbances have ...Understanding the neural underpinning of human gait and balance is one of the most pertinent challenges for 21st-century translational neuroscience due to the profound impact that falls and mobility disturbances have on our aging population.Posture and gait control does not happen automatically,as previously believed,but rather requires continuous involvement of central nervous mechanisms.To effectively exert control over the body,the brain must integrate multiple streams of sensory information,including visual,vestibular,and somatosensory signals.The mechanisms which underpin the integration of these multisensory signals are the principal topic of the present work.Existing multisensory integration theories focus on how failure of cognitive processes thought to be involved in multisensory integration leads to falls in older adults.Insufficient emphasis,however,has been placed on specific contributions of individual sensory modalities to multisensory integration processes and cross-modal interactions that occur between the sensory modalities in relation to gait and balance.In the present work,we review the contributions of somatosensory,visual,and vestibular modalities,along with their multisensory intersections to gait and balance in older adults and patients with Parkinson’s disease.We also review evidence of vestibular contributions to multisensory temporal binding windows,previously shown to be highly pertinent to fall risk in older adults.Lastly,we relate multisensory vestibular mechanisms to potential neural substrates,both at the level of neurobiology(concerning positron emission tomography imaging)and at the level of electrophysiology(concerning electroencephalography).We hope that this integrative review,drawing influence across multiple subdisciplines of neuroscience,paves the way for novel research directions and therapeutic neuromodulatory approaches,to improve the lives of older adults and patients with neurodegenerative diseases.展开更多
Background Exploring how immersive technologies can simulate and assess user experiences in designed environments is an important topic in architectural research.In this study,a multisensory virtual reality(VR)system ...Background Exploring how immersive technologies can simulate and assess user experiences in designed environments is an important topic in architectural research.In this study,a multisensory virtual reality(VR)system developed to support the study of human-built environment interactions under multimodal conditions(visual,olfactory,and auditory)was evaluated.Methods The effectiveness of the system was tested by conducting in-depth user studies using a mixed-method approach to provide quantitative and qualitative evidence.The results of the case study were discussed,key features of the proposed prototype were assessed,and limitations and opportunities for future studies were identified.Results Findings showed that multisensory elements can deepen participants’sense of presence,increase engagement levels,and enrich overall user experience in immersive environments.Integrating olfactory stimuli into virtual representations of architectural spaces revealed how multisensory feedback informs spatial perception and supports the development of more responsive and human-centered design strategies.Conclusions This study contributes to the emerging field of sensory architecture,aiming to move beyond visual simulation toward a richer embodied understanding of space.The proposed approach provides valuable insights into the development of multisensory VR environments in architecture,enabling future research and immersive research methodologies in wider fields.展开更多
Artificial multisensory devices play a key role in human-computer interaction in the field of artificial intelligence(AI).In this work,we have designed and constructed a novel olfactory-visual bimodal neuromorphic car...Artificial multisensory devices play a key role in human-computer interaction in the field of artificial intelligence(AI).In this work,we have designed and constructed a novel olfactory-visual bimodal neuromorphic carbon nanotube thin film transistor(TFT)arrays for artificial olfactory-visual multisensory synergy recognition with a very low power consumption of 25 aJ for a single pulse,employing semiconducting single-walled carbon nanotubes(sc-SWCNTs)as channel materials and gas sensitive materials,and poly[[4,8-bis[5-(2-ethylhexyl)-2-thienyl]benzo[1,2-b:4,5-b0]dithiophene-2,6-diyl]-2,5-thiophenediyl-[5,7-bis(2-ethylhexyl)-4,8-dioxo-4H,8H-benzo[1,2-c:4,5-c0]dithio-phene-1,3-diyl]](PBDB-T)as the photosensitive material.It is noted that it is the first time to realize the simulation of olfactory and visual senses(from 280 nm to 650 nm)with the wide operating temperature range(0-150℃)in a single SWCNT TFT device and successfully simulate the recovery of olfactory senses after COVID-19 by olfactory-visual synergy.Furthermore,our SWCNT neuromorphic TFT devices with a high IOn/IOff ratio(up to 10^(6))at a low operating voltage(−2 to 0.5 V)can mimic not only the basic biological synaptic functions of olfaction and vision(such as paired-pulse facilitation,short-term plasticity,and long-term plasticity),but also optical wireless communication by Morse code.The proposed multisensory,broadband light-responsive,low-power synaptic devices provide great potential for developing AI robots to face complex external environments.展开更多
Objectives:Emotional stress is a significant public health challenge.Virtual reality(VR)offers the potential for aiding emotional recovery.This study explores the impact of VR environment design factors on emotional r...Objectives:Emotional stress is a significant public health challenge.Virtual reality(VR)offers the potential for aiding emotional recovery.This study explores the impact of VR environment design factors on emotional recovery,examining underlying mechanisms through physiological indicators and behavioral responses.Methods:Two experimentswere conducted.Experiment 1 employed a 4[Scene Type:real environment(RE),virtual scenes that restore the RE(VR),virtual scenes that incorporate natural window view design(VR-W),and a no-scene control condition(CTL)]×3(Experimental Phase:baseline,emotion arousal,recovery)mixed design(N=33).Participants viewed a 4-min anxiety-inducing video followed by a 3-min scene exposure.State-Trait Anxiety Inventory-State Form(STAIS),galvanic skin response(GSR),and blood-volume pulse(BVP)frequency were analyzed with linear mixed-effects models.Experiment 2 used a 3(Motion-control mode:Unnatural,Semi-natural,Natural)×2(Sound form:Spatial positioning,Surround)×3(Experimental Phase:baseline,emotion arousal,recovery)mixed design(N=42).Presence was analyzed with the Scheirer–Ray–Hare test;phase efficacy was verified with Friedman tests.Results:Experiment 1 showed significant Scene Type×Experimental Phase interactions for GSR(F=8.006,p<0.001,η^(2)_(p)=0.624)and BVP frequency(F=11.491,p<0.001,η^(2)_(p)=0.704).VR-Wproduced the largest recovery(ΔGSR=–1.26;ΔBVP=–5.80;Hedges g≥0.83)vs.RE and VR.STAI-S returned to baseline across all Scene Types.Experiment 2 revealedmain effects of Motion-control mode(F=8.55,p=0.001,η^(2)_(p)=0.32)and Sound form(F=4.35,p=0.044,η^(2)_(p)=0.11)on Presence(Semi-natural+Spatial positioning highest).The greatest physiological recovery occurred with the UnnaturalMotioncontrolmode(GSRH=20.17,p<0.001,ε^(2)=0.49;BVPH=7.92,p=0.019),amplified by Spatial positioning Sound form only in this mode.Design factors did not influence STAI-S change.Conclusions:VR scenes are as restorative as RE;embedding VR-W accelerates recovery.Maximal Presence is not essential:Unnatural Motion-control mode induced the largest physiological recovery,especially combined with Spatial positioning Sound form.展开更多
Over the last decade virtual reality （VR） setups for rodents have been developed and utilized to in-vestigate the neural foundations of behavior. Such VR systems became very popular since theyallow the use of state-...Over the last decade virtual reality （VR） setups for rodents have been developed and utilized to in-vestigate the neural foundations of behavior. Such VR systems became very popular since theyallow the use of state-of-the-art techniques to measure neural activity in behaving rodents that can-not be easily used with classical behavior setups. Here, we provide an overview of rodent VR tech-nologies and review recent results from related research. We discuss commonalities and differ-ences as well as merits and issues of different approaches. A special focus is given to experimental（behavioral） paradigms in use. Finally we comment on possible use cases that may further exploitthe potential of VR in rodent research and hence inspire future studies.展开更多
Urbanization of animal habitats has the potential to affect the natural communication systems of any species able to survive in the changed environment. Urban animals such as squirrels use multiple signal channels to ...Urbanization of animal habitats has the potential to affect the natural communication systems of any species able to survive in the changed environment. Urban animals such as squirrels use multiple signal channels to communicate, but it is un- known how ttrbanization has affected these behaviors. Multimodal commtmication, involving more than one sensory modality, can be studied by use of biomimetic mechanical animal models that are designed to simulate the multimodal signals and be pre- sented to animal subjects in the field. In this way the responses to the various signal components can be compared and contrasted to determine whether the multimodal signal is made up of redundant or nonredundant components. In this study, we presented wild gray squirrels in relatively urban and relatively rural habitats in Western Massachusetts with a biomimetic squirrel model that produced tail flags and alarm barks in a variety of combinations. We found that the squirrels responded to each unimodal component on its own, the bark and tail flag, but they responded most to the complete multimodal signal, containing both the acoustic and the moving visual components, providing evidence that in this context the signal components are redundant and that their combination elicits multimodal enhancement. We expanded on the results of Partan et al. （2009） by providing data on sig- naling behavior in the presence and absence of conspecifics, suggesting that alarm signaling is more likely if conspecifics are present. We found that the squirrels were more active in the urban habitats and that they responded more to tail flagging in the urban habitats as compared to the rural ones, suggesting the interesting possibility of a multimodal shift from reliance on audio to visual signals in noisier more crowded urban habitats [Current Zoology 56 （3）： 313-326, 2010].展开更多
In the robotic community more and more hands are developed. Based on theexperience of HIT Hand and DLR Hand II, a smaller and easier manufactured dexterous robot hand withmultisen-sory function and high integration is...In the robotic community more and more hands are developed. Based on theexperience of HIT Hand and DLR Hand II, a smaller and easier manufactured dexterous robot hand withmultisen-sory function and high integration is jointly developed. The prototype of the hand issuccessfully built. It has 4 fingers in total 13-DOFs (degree of freedom). Each finger has 3-DOFsand 4 joints, the last 2 joints are mechanically coupled by means of four-bar linkage mechanism. Italso has an additional DOF to realize motion of the thumb relative to the palm. The fingertip forcecan reach up to 10 N. Full integration of mechanical body, actuation system, multisensory system andelectronics is a significant feature. DSP based control system is implemented in PCI busarchitecture and the serial communication between the hand and DSP needs only 2 lines.展开更多
Multisensory enhancement,as a facilitation phenomenon,is responsible for superior behavioral performance when an individual is responding to cross-modal versus modality-specific stimuli.However,the event-related poten...Multisensory enhancement,as a facilitation phenomenon,is responsible for superior behavioral performance when an individual is responding to cross-modal versus modality-specific stimuli.However,the event-related potential（ERP） counterparts of behavioral multisensory enhancement are not well known.We recorded ERPs and behavioral data from 14 healthy volunteers with three types of target stimuli（modality-specific,bimodal,and trimodal） to examine the spatio-temporal electrophysiological characteristics of multisensory enhancement by comparing behavioral data with ERPs.We found a strong correlation between P3 latency and behavioral performance in terms of reaction time（RT）（R = 0.98,P ＆lt;0.001）,suggesting that P3 latency constitutes a temporal measure of behavioral multisensory enhancement.In addition,a fast RT and short P3 latency were found when comparing the modality-specific visual target with the modality-specific auditory and somatosensory targets.Our results indicate that behavioral multisensory enhancement can be identified by the latency and source distribution of the P3 component.These findings may advance our understanding of the neuronal mechanisms of multisensory enhancement.展开更多
This paper presents a new developed anthropomorphic robot dexterous hand： HIT/DLR Hand II. The hand is composed of an independent palm and five identical modular fingers, and each finger has three degree of freedom ...This paper presents a new developed anthropomorphic robot dexterous hand： HIT/DLR Hand II. The hand is composed of an independent palm and five identical modular fingers, and each finger has three degree of freedom （DOFs） and four joints. All the actuators and electronics are integrated in the finger body and the palm. Owing to using a new actuator, drivers and a novel arrangement, both the length and width of the finger is about two third of its formner version. By using the wire coupling mecha- nism, the distal phalanx transmission ratio is kept exactly 1 ： 1 in the whole movement range. The packing mechanism which is implemented directly in the finger body and palm not only reduces the size of whole hand but also make it more anthropomorphic. Additionally, the new designed force/torque and position sensors are integrated in the hand for increasing muhisensory capability. To evaluate the performances of the finger mechanism, the position and impedance control experiments are conducted.展开更多
Accurate self-motion perception,which is critical for organisms to survive,is a process involving multiple sensory cues.The two most powerful cues are visual(optic flow)and vestibular(inertial motion).Psychophysical s...Accurate self-motion perception,which is critical for organisms to survive,is a process involving multiple sensory cues.The two most powerful cues are visual(optic flow)and vestibular(inertial motion).Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction,often in a statistically Bayesian-optimal way.In the last decade,single-unit recordings in awake,behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution,giving insight into possible neural mechanisms underlying multisensory self-motion perception.Here,we review these findings,along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities.We show that,in light of new data,conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged.We propose that different temporal component signals may mediate different functions,a possibility that requires future studies.展开更多
Cross-modal semantic mapping and cross-media retrieval are key problems of the multimedia search engine.This study analyzes the hierarchy,the functionality,and the structure in the visual and auditory sensations of co...Cross-modal semantic mapping and cross-media retrieval are key problems of the multimedia search engine.This study analyzes the hierarchy,the functionality,and the structure in the visual and auditory sensations of cognitive system,and establishes a brain-like cross-modal semantic mapping framework based on cognitive computing of visual and auditory sensations.The mechanism of visual-auditory multisensory integration,selective attention in thalamo-cortical,emotional control in limbic system and the memory-enhancing in hippocampal were considered in the framework.Then,the algorithms of cross-modal semantic mapping were given.Experimental results show that the framework can be effectively applied to the cross-modal semantic mapping,and also provides an important significance for brain-like computing of non-von Neumann structure.展开更多
The present study observed the effects of the non-invasive, mirror neuron system application on neurorehabilitation. In primate studies, mirror neurons have been shown to fire when the subject observes or performs a s...The present study observed the effects of the non-invasive, mirror neuron system application on neurorehabilitation. In primate studies, mirror neurons have been shown to fire when the subject observes or performs a specific action, thereby allowing for observation of motor cortex activation. This activation of the mirror neuron system could serve as a treatment for stroke patients. In the present study, the combination of a mirror neuron system-based therapy was introduced for the treatment of patients with motor-deficits, who could not perform rehabilitation exercises. The results also indicate that this therapeutic method plays a positive role in emotional regulation in the same patients.展开更多
Progress in understanding multisensory integration in human have suggested researchers that the integration may result into the enhancement or depression of incoming signals.It is evident based on different psychologi...Progress in understanding multisensory integration in human have suggested researchers that the integration may result into the enhancement or depression of incoming signals.It is evident based on different psychological and behavioral experiments that stimuli coming from different perceptual modalities at the same time or from the same place,the signal having more strength under the influence of emotions effects the response accordingly.Current research inmultisensory integration has not studied the effect of emotions despite its significance and natural influence in multisensory enhancement or depression.Therefore,there is a need to integrate the emotional state of the agent with incoming stimuli for signal enhancement or depression.In this study,two different neural network-based learning algorithms have been employed to learn the impact of emotions on signal enhancement or depression.It was observed that the performance of a proposed system for multisensory integration increases when emotion features were present during enhancement or depression of multisensory signals.展开更多
This paper presents a novel remote controlled dexterous robot arm with 6 degrees of freedom （DOF）. As a highly integrated mechatronics system, sensors and their signal processing system are integrated inside each jo...This paper presents a novel remote controlled dexterous robot arm with 6 degrees of freedom （DOF）. As a highly integrated mechatronics system, sensors and their signal processing system are integrated inside each joint. To lighten the weight, almost all mechanical parts are made of aluminum and the robot control system is placed outside. The modular concept is adopted during the robot design process for time and cost saving. Considering the much greater torque acted on the two shoulder joints, the joint shells are strengthened in the design to increase joint stiffness and suppress system vibration. Meanwhile, to simplify the maintenance, a new spring pins electronic connector is designed to disassemble every joint, connector and link independently without cutting any cables. The teleoperation technology enables the robot to offer more convenient service definitely for people＇ s daily life. Virtual reality technology is used to solve the time delay problem during teleoperation. Finally, two typical daily chore experiments are implemented to prove the manipulation ability of the dexterous robot arm.展开更多
Energy consumption of the EU has a crucial environmental impact;several efforts are nowadays thus directed into massively reducing energy consumption by envelope improvement,system efficiency and smart control.On the ...Energy consumption of the EU has a crucial environmental impact;several efforts are nowadays thus directed into massively reducing energy consumption by envelope improvement,system efficiency and smart control.On the other hand,the indoor thermal and lighting conditions significantly influence users’wellbeing and productivity,which is especially important when dealing with educational and working facilities.Strategies to enhance system efficiency are focused on design and construction aspects.These strategies ease to promote a powerful approach which is needed when focusing on existing buildings in need of retrofit measures.When dealing with new or refurbished buildings,energy saving has a further step to achieve.In the last years,research trends moved towards an increasing inclusion of human factors in energy evaluation.This allows to account for the occupancy variability in the energy analyses,considering how to bridge the performance gap between predictive models and actual consumptions due to indoor thermal settings.In empty buildings energy consumption is huge and economic efforts are wasted,due to unconscious energy-wasting behaviors.Previous studies with Interactive Virtual Environments confirmed that indoor environments illuminated by different color lights lead to the perception of different levels of thermal comfort.The results of the present paper replicate previous evidence collected in real conditions,suggesting that Virtual Reality is a valid and reliable tool to assess thermal comfort more quickly and cheaply.This paper provides a further perspective on this topic,including also the use of different fragrances to understand how the indoor environment could be enhanced and manipulated to increase wellbeing,thermal perception and finally energy saving.展开更多
Background:The ability to track objects as they move is critical for successful interaction with objects in the world.The multiple object tracking(MOT)paradigm has demonstrated that,within limits,our visual attention ...Background:The ability to track objects as they move is critical for successful interaction with objects in the world.The multiple object tracking(MOT)paradigm has demonstrated that,within limits,our visual attention capacity allows us to track multiple moving objects among distracters.Very little is known about dynamic auditory attention and the role of multisensory binding in attentional tracking.Here,we examined whether dynamic sounds congruent with visual targets could facilitate tracking in a 3D-MOT task.Methods:Participants tracked one or multiple target-spheres among identical distractor-spheres during 8 seconds of movement in a virtual cube.In the visual condition,targets were identified with a brief colour change,but were then indistinguishable from the distractors during the movement.In the audio-visual condition,the target-spheres were accompanied by a sound,which moved congruently with the change in the target’s position.Sound amplitude varied with distance from the observer and inter-aural amplitude difference varied with azimuth.Results:Results with one target showed that performance was better in the audiovisual condition,which suggests that congruent sounds can facilitate attentional visual tracking.However,with multiple targets,the sounds did not facilitate tracking.Conclusions:This suggests that audiovisual binding may not be possible when attention is divided between multiple targets.展开更多
Background:Saccades are rapid and abrupt eye movements that allow us to change the point of fixation very quickly.Saccades are generally made to visual points of interest,but we can also saccade to non-visual objects ...Background:Saccades are rapid and abrupt eye movements that allow us to change the point of fixation very quickly.Saccades are generally made to visual points of interest,but we can also saccade to non-visual objects that attract our attention.While there is a plethora of studies investigating saccadic eye movements to visual targets,there is very little evidence of how eye movement planning occurs when individuals are performing eye movements to non-visual targets across different sensory modalities.Methods:Fifteen adults with normal,or corrected to normal,vision made saccades to either visual,auditory,tactile or proprioceptive targets.In the auditory condition a speaker was positioned at one of eight locations along a circle surrounding a central fixation point.In the proprioceptive condition the participant’s finger was placed at one of the eight locations.In the tactile condition participants were touched on their right forearm in one of four eccentric location,left and right of a central point.Eye movements were made in complete darkness.Results:We compared the precision and accuracy of the eye movements to tactile,proprioceptive,and auditory targets in the dark.Overall,both precision and accuracy of movements to non-visual targets were significantly lower compared to visual targets.Conclusions:These differences emphasize the central role of the visual system in saccade planning.展开更多
基金the financial support from the National Key Research and Development Program of China(Grant No.2022YFB4400100)the NSFC under Grant Nos.92477102 and 62122084the open research fund of Songshan Lake Materials Laboratory 2023SLABFK09。
文摘The increasing complexity of intelligent sensing environments,driven by the growth of Internet of Things technologies,has created a strong demand for neuromorphic systems capable of real-time,low-power multisensory perception.Traditional sensory architectures,constrained by single-modal processing and centralized computing,struggle to meet the requirements of diverse and dynamic input conditions.Multisensory neuromorphic devices offer a promising solution by mimicking the distributed,event-driven processing of biological systems.Recent efforts have explored synaptic devices and material systems that respond to various input modalities,including visual,tactile,thermal,and chemical stimuli.However,challenges remain in signal conversion,encoding compatibility,and the fusion of heterogeneous inputs without loss of unisensory information.This review provides a comprehensive overview of the physical mechanisms,device behaviors,and integration strategies that underpin signal processing in neuromorphic hardware.We highlight synaptic mechanisms conducive to cross-modal interaction,analyze representative signal fusion approaches at the device level,and discuss future directions for constructing efficient,scalable,and biologically inspired multisensory neuromorphic systems.
文摘Understanding the neural underpinning of human gait and balance is one of the most pertinent challenges for 21st-century translational neuroscience due to the profound impact that falls and mobility disturbances have on our aging population.Posture and gait control does not happen automatically,as previously believed,but rather requires continuous involvement of central nervous mechanisms.To effectively exert control over the body,the brain must integrate multiple streams of sensory information,including visual,vestibular,and somatosensory signals.The mechanisms which underpin the integration of these multisensory signals are the principal topic of the present work.Existing multisensory integration theories focus on how failure of cognitive processes thought to be involved in multisensory integration leads to falls in older adults.Insufficient emphasis,however,has been placed on specific contributions of individual sensory modalities to multisensory integration processes and cross-modal interactions that occur between the sensory modalities in relation to gait and balance.In the present work,we review the contributions of somatosensory,visual,and vestibular modalities,along with their multisensory intersections to gait and balance in older adults and patients with Parkinson’s disease.We also review evidence of vestibular contributions to multisensory temporal binding windows,previously shown to be highly pertinent to fall risk in older adults.Lastly,we relate multisensory vestibular mechanisms to potential neural substrates,both at the level of neurobiology(concerning positron emission tomography imaging)and at the level of electrophysiology(concerning electroencephalography).We hope that this integrative review,drawing influence across multiple subdisciplines of neuroscience,paves the way for novel research directions and therapeutic neuromodulatory approaches,to improve the lives of older adults and patients with neurodegenerative diseases.
文摘Background Exploring how immersive technologies can simulate and assess user experiences in designed environments is an important topic in architectural research.In this study,a multisensory virtual reality(VR)system developed to support the study of human-built environment interactions under multimodal conditions(visual,olfactory,and auditory)was evaluated.Methods The effectiveness of the system was tested by conducting in-depth user studies using a mixed-method approach to provide quantitative and qualitative evidence.The results of the case study were discussed,key features of the proposed prototype were assessed,and limitations and opportunities for future studies were identified.Results Findings showed that multisensory elements can deepen participants’sense of presence,increase engagement levels,and enrich overall user experience in immersive environments.Integrating olfactory stimuli into virtual representations of architectural spaces revealed how multisensory feedback informs spatial perception and supports the development of more responsive and human-centered design strategies.Conclusions This study contributes to the emerging field of sensory architecture,aiming to move beyond visual simulation toward a richer embodied understanding of space.The proposed approach provides valuable insights into the development of multisensory VR environments in architecture,enabling future research and immersive research methodologies in wider fields.
基金supported by the National Key Research and Development Program of China(2020YFA0714700)Natural Science Foundation of China(62274174)+3 种基金Key Research and Development Program of Jiangsu Province(BK20232009)a fellowship from the China Postdoctoral Science Foundation(NO:2023M742559)the Cooperation Project of Vacuum Interconnect Research Facility(NANO-X)of Suzhou Institute of Nano-Tech and Nano-Bionics,Chinese Academy of Sciences(F2208)the technical support for Nano-X from Suzhou Institute of Nano-Tech and Nano-Bionics,Chinese Academy of Sciences(SINANO)。
文摘Artificial multisensory devices play a key role in human-computer interaction in the field of artificial intelligence(AI).In this work,we have designed and constructed a novel olfactory-visual bimodal neuromorphic carbon nanotube thin film transistor(TFT)arrays for artificial olfactory-visual multisensory synergy recognition with a very low power consumption of 25 aJ for a single pulse,employing semiconducting single-walled carbon nanotubes(sc-SWCNTs)as channel materials and gas sensitive materials,and poly[[4,8-bis[5-(2-ethylhexyl)-2-thienyl]benzo[1,2-b:4,5-b0]dithiophene-2,6-diyl]-2,5-thiophenediyl-[5,7-bis(2-ethylhexyl)-4,8-dioxo-4H,8H-benzo[1,2-c:4,5-c0]dithio-phene-1,3-diyl]](PBDB-T)as the photosensitive material.It is noted that it is the first time to realize the simulation of olfactory and visual senses(from 280 nm to 650 nm)with the wide operating temperature range(0-150℃)in a single SWCNT TFT device and successfully simulate the recovery of olfactory senses after COVID-19 by olfactory-visual synergy.Furthermore,our SWCNT neuromorphic TFT devices with a high IOn/IOff ratio(up to 10^(6))at a low operating voltage(−2 to 0.5 V)can mimic not only the basic biological synaptic functions of olfaction and vision(such as paired-pulse facilitation,short-term plasticity,and long-term plasticity),but also optical wireless communication by Morse code.The proposed multisensory,broadband light-responsive,low-power synaptic devices provide great potential for developing AI robots to face complex external environments.
基金funded by the General Project of the National Social Science Fund(NSSF)of China in Art(Project No.23BG133)the Scientific Research Foundation ofWuhan Institute of Technology(Project No.K2023064)+2 种基金the Open Fund of Key Research Base of Humanities and Social Sciences in Hubei Province—Research Center for College Students’Development and Innovation Education:“VR Healing Intervention for College Students’Academic Anxiety—VR Healing Based on Intention Dialogue”(Project No.DXS2023017)the Research Project of China Youth and Children Research Society(Project No.2025B11)the Major Strategic Consulting Projects of the Chinese Academy of Engineering between Academy and Local Regions(Project No.2024-DFZD-41).
文摘Objectives:Emotional stress is a significant public health challenge.Virtual reality(VR)offers the potential for aiding emotional recovery.This study explores the impact of VR environment design factors on emotional recovery,examining underlying mechanisms through physiological indicators and behavioral responses.Methods:Two experimentswere conducted.Experiment 1 employed a 4[Scene Type:real environment(RE),virtual scenes that restore the RE(VR),virtual scenes that incorporate natural window view design(VR-W),and a no-scene control condition(CTL)]×3(Experimental Phase:baseline,emotion arousal,recovery)mixed design(N=33).Participants viewed a 4-min anxiety-inducing video followed by a 3-min scene exposure.State-Trait Anxiety Inventory-State Form(STAIS),galvanic skin response(GSR),and blood-volume pulse(BVP)frequency were analyzed with linear mixed-effects models.Experiment 2 used a 3(Motion-control mode:Unnatural,Semi-natural,Natural)×2(Sound form:Spatial positioning,Surround)×3(Experimental Phase:baseline,emotion arousal,recovery)mixed design(N=42).Presence was analyzed with the Scheirer–Ray–Hare test;phase efficacy was verified with Friedman tests.Results:Experiment 1 showed significant Scene Type×Experimental Phase interactions for GSR(F=8.006,p<0.001,η^(2)_(p)=0.624)and BVP frequency(F=11.491,p<0.001,η^(2)_(p)=0.704).VR-Wproduced the largest recovery(ΔGSR=–1.26;ΔBVP=–5.80;Hedges g≥0.83)vs.RE and VR.STAI-S returned to baseline across all Scene Types.Experiment 2 revealedmain effects of Motion-control mode(F=8.55,p=0.001,η^(2)_(p)=0.32)and Sound form(F=4.35,p=0.044,η^(2)_(p)=0.11)on Presence(Semi-natural+Spatial positioning highest).The greatest physiological recovery occurred with the UnnaturalMotioncontrolmode(GSRH=20.17,p<0.001,ε^(2)=0.49;BVPH=7.92,p=0.019),amplified by Spatial positioning Sound form only in this mode.Design factors did not influence STAI-S change.Conclusions:VR scenes are as restorative as RE;embedding VR-W accelerates recovery.Maximal Presence is not essential:Unnatural Motion-control mode induced the largest physiological recovery,especially combined with Spatial positioning Sound form.
文摘Over the last decade virtual reality （VR） setups for rodents have been developed and utilized to in-vestigate the neural foundations of behavior. Such VR systems became very popular since theyallow the use of state-of-the-art techniques to measure neural activity in behaving rodents that can-not be easily used with classical behavior setups. Here, we provide an overview of rodent VR tech-nologies and review recent results from related research. We discuss commonalities and differ-ences as well as merits and issues of different approaches. A special focus is given to experimental（behavioral） paradigms in use. Finally we comment on possible use cases that may further exploitthe potential of VR in rodent research and hence inspire future studies.
文摘Urbanization of animal habitats has the potential to affect the natural communication systems of any species able to survive in the changed environment. Urban animals such as squirrels use multiple signal channels to communicate, but it is un- known how ttrbanization has affected these behaviors. Multimodal commtmication, involving more than one sensory modality, can be studied by use of biomimetic mechanical animal models that are designed to simulate the multimodal signals and be pre- sented to animal subjects in the field. In this way the responses to the various signal components can be compared and contrasted to determine whether the multimodal signal is made up of redundant or nonredundant components. In this study, we presented wild gray squirrels in relatively urban and relatively rural habitats in Western Massachusetts with a biomimetic squirrel model that produced tail flags and alarm barks in a variety of combinations. We found that the squirrels responded to each unimodal component on its own, the bark and tail flag, but they responded most to the complete multimodal signal, containing both the acoustic and the moving visual components, providing evidence that in this context the signal components are redundant and that their combination elicits multimodal enhancement. We expanded on the results of Partan et al. （2009） by providing data on sig- naling behavior in the presence and absence of conspecifics, suggesting that alarm signaling is more likely if conspecifics are present. We found that the squirrels were more active in the urban habitats and that they responded more to tail flagging in the urban habitats as compared to the rural ones, suggesting the interesting possibility of a multimodal shift from reliance on audio to visual signals in noisier more crowded urban habitats [Current Zoology 56 （3）： 313-326, 2010].
文摘In the robotic community more and more hands are developed. Based on theexperience of HIT Hand and DLR Hand II, a smaller and easier manufactured dexterous robot hand withmultisen-sory function and high integration is jointly developed. The prototype of the hand issuccessfully built. It has 4 fingers in total 13-DOFs (degree of freedom). Each finger has 3-DOFsand 4 joints, the last 2 joints are mechanically coupled by means of four-bar linkage mechanism. Italso has an additional DOF to realize motion of the thumb relative to the palm. The fingertip forcecan reach up to 10 N. Full integration of mechanical body, actuation system, multisensory system andelectronics is a significant feature. DSP based control system is implemented in PCI busarchitecture and the serial communication between the hand and DSP needs only 2 lines.
基金partly supported by the National Natural Science Foundation of China(81271685)Key Project of Science and Technology Fund of Tianjin Municipality,China(10JCZDJC16100)+1 种基金the China Postdoctoral Science Foundation(2012M510754)the Independent Innovative Fund of Tianjin University,Tianjin Municipality,China(1102)
文摘Multisensory enhancement,as a facilitation phenomenon,is responsible for superior behavioral performance when an individual is responding to cross-modal versus modality-specific stimuli.However,the event-related potential（ERP） counterparts of behavioral multisensory enhancement are not well known.We recorded ERPs and behavioral data from 14 healthy volunteers with three types of target stimuli（modality-specific,bimodal,and trimodal） to examine the spatio-temporal electrophysiological characteristics of multisensory enhancement by comparing behavioral data with ERPs.We found a strong correlation between P3 latency and behavioral performance in terms of reaction time（RT）（R = 0.98,P ＆lt;0.001）,suggesting that P3 latency constitutes a temporal measure of behavioral multisensory enhancement.In addition,a fast RT and short P3 latency were found when comparing the modality-specific visual target with the modality-specific auditory and somatosensory targets.Our results indicate that behavioral multisensory enhancement can be identified by the latency and source distribution of the P3 component.These findings may advance our understanding of the neuronal mechanisms of multisensory enhancement.
基金supported by the National High Technology Research and Development Programme of China(2006AA04Z255)the 111 Project(B307018)
文摘This paper presents a new developed anthropomorphic robot dexterous hand： HIT/DLR Hand II. The hand is composed of an independent palm and five identical modular fingers, and each finger has three degree of freedom （DOFs） and four joints. All the actuators and electronics are integrated in the finger body and the palm. Owing to using a new actuator, drivers and a novel arrangement, both the length and width of the finger is about two third of its formner version. By using the wire coupling mecha- nism, the distal phalanx transmission ratio is kept exactly 1 ： 1 in the whole movement range. The packing mechanism which is implemented directly in the finger body and palm not only reduces the size of whole hand but also make it more anthropomorphic. Additionally, the new designed force/torque and position sensors are integrated in the hand for increasing muhisensory capability. To evaluate the performances of the finger mechanism, the position and impedance control experiments are conducted.
基金supported by grants from the National Science and Technology Innovation 2030 Major Program(2022ZD0205000)the Strategic Priority Research Program of CAS(XDB32070000)+1 种基金the Shanghai Municipal Science and Technology Major Project(2018SHZDZX05)the Shanghai Academic Research Leader Program(21XD1404000).
文摘Accurate self-motion perception,which is critical for organisms to survive,is a process involving multiple sensory cues.The two most powerful cues are visual(optic flow)and vestibular(inertial motion).Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction,often in a statistically Bayesian-optimal way.In the last decade,single-unit recordings in awake,behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution,giving insight into possible neural mechanisms underlying multisensory self-motion perception.Here,we review these findings,along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities.We show that,in light of new data,conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged.We propose that different temporal component signals may mediate different functions,a possibility that requires future studies.
基金Supported by the National Natural Science Foundation of China(No.61305042,61202098)Projects of Center for Remote Sensing Mission Study of China National Space Administration(No.2012A03A0939)Science and Technological Research of Key Projects of Education Department of Henan Province of China(No.13A520071)
文摘Cross-modal semantic mapping and cross-media retrieval are key problems of the multimedia search engine.This study analyzes the hierarchy,the functionality,and the structure in the visual and auditory sensations of cognitive system,and establishes a brain-like cross-modal semantic mapping framework based on cognitive computing of visual and auditory sensations.The mechanism of visual-auditory multisensory integration,selective attention in thalamo-cortical,emotional control in limbic system and the memory-enhancing in hippocampal were considered in the framework.Then,the algorithms of cross-modal semantic mapping were given.Experimental results show that the framework can be effectively applied to the cross-modal semantic mapping,and also provides an important significance for brain-like computing of non-von Neumann structure.
文摘The present study observed the effects of the non-invasive, mirror neuron system application on neurorehabilitation. In primate studies, mirror neurons have been shown to fire when the subject observes or performs a specific action, thereby allowing for observation of motor cortex activation. This activation of the mirror neuron system could serve as a treatment for stroke patients. In the present study, the combination of a mirror neuron system-based therapy was introduced for the treatment of patients with motor-deficits, who could not perform rehabilitation exercises. The results also indicate that this therapeutic method plays a positive role in emotional regulation in the same patients.
基金This work was supported by the GRRC program of Gyeonggi province.[GRRC-Gachon2020(B04),Development of AI-based Healthcare Devices].
文摘Progress in understanding multisensory integration in human have suggested researchers that the integration may result into the enhancement or depression of incoming signals.It is evident based on different psychological and behavioral experiments that stimuli coming from different perceptual modalities at the same time or from the same place,the signal having more strength under the influence of emotions effects the response accordingly.Current research inmultisensory integration has not studied the effect of emotions despite its significance and natural influence in multisensory enhancement or depression.Therefore,there is a need to integrate the emotional state of the agent with incoming stimuli for signal enhancement or depression.In this study,two different neural network-based learning algorithms have been employed to learn the impact of emotions on signal enhancement or depression.It was observed that the performance of a proposed system for multisensory integration increases when emotion features were present during enhancement or depression of multisensory signals.
文摘This paper presents a novel remote controlled dexterous robot arm with 6 degrees of freedom （DOF）. As a highly integrated mechatronics system, sensors and their signal processing system are integrated inside each joint. To lighten the weight, almost all mechanical parts are made of aluminum and the robot control system is placed outside. The modular concept is adopted during the robot design process for time and cost saving. Considering the much greater torque acted on the two shoulder joints, the joint shells are strengthened in the design to increase joint stiffness and suppress system vibration. Meanwhile, to simplify the maintenance, a new spring pins electronic connector is designed to disassemble every joint, connector and link independently without cutting any cables. The teleoperation technology enables the robot to offer more convenient service definitely for people＇ s daily life. Virtual reality technology is used to solve the time delay problem during teleoperation. Finally, two typical daily chore experiments are implemented to prove the manipulation ability of the dexterous robot arm.
文摘Energy consumption of the EU has a crucial environmental impact;several efforts are nowadays thus directed into massively reducing energy consumption by envelope improvement,system efficiency and smart control.On the other hand,the indoor thermal and lighting conditions significantly influence users’wellbeing and productivity,which is especially important when dealing with educational and working facilities.Strategies to enhance system efficiency are focused on design and construction aspects.These strategies ease to promote a powerful approach which is needed when focusing on existing buildings in need of retrofit measures.When dealing with new or refurbished buildings,energy saving has a further step to achieve.In the last years,research trends moved towards an increasing inclusion of human factors in energy evaluation.This allows to account for the occupancy variability in the energy analyses,considering how to bridge the performance gap between predictive models and actual consumptions due to indoor thermal settings.In empty buildings energy consumption is huge and economic efforts are wasted,due to unconscious energy-wasting behaviors.Previous studies with Interactive Virtual Environments confirmed that indoor environments illuminated by different color lights lead to the perception of different levels of thermal comfort.The results of the present paper replicate previous evidence collected in real conditions,suggesting that Virtual Reality is a valid and reliable tool to assess thermal comfort more quickly and cheaply.This paper provides a further perspective on this topic,including also the use of different fragrances to understand how the indoor environment could be enhanced and manipulated to increase wellbeing,thermal perception and finally energy saving.
文摘Background:The ability to track objects as they move is critical for successful interaction with objects in the world.The multiple object tracking(MOT)paradigm has demonstrated that,within limits,our visual attention capacity allows us to track multiple moving objects among distracters.Very little is known about dynamic auditory attention and the role of multisensory binding in attentional tracking.Here,we examined whether dynamic sounds congruent with visual targets could facilitate tracking in a 3D-MOT task.Methods:Participants tracked one or multiple target-spheres among identical distractor-spheres during 8 seconds of movement in a virtual cube.In the visual condition,targets were identified with a brief colour change,but were then indistinguishable from the distractors during the movement.In the audio-visual condition,the target-spheres were accompanied by a sound,which moved congruently with the change in the target’s position.Sound amplitude varied with distance from the observer and inter-aural amplitude difference varied with azimuth.Results:Results with one target showed that performance was better in the audiovisual condition,which suggests that congruent sounds can facilitate attentional visual tracking.However,with multiple targets,the sounds did not facilitate tracking.Conclusions:This suggests that audiovisual binding may not be possible when attention is divided between multiple targets.
文摘Background:Saccades are rapid and abrupt eye movements that allow us to change the point of fixation very quickly.Saccades are generally made to visual points of interest,but we can also saccade to non-visual objects that attract our attention.While there is a plethora of studies investigating saccadic eye movements to visual targets,there is very little evidence of how eye movement planning occurs when individuals are performing eye movements to non-visual targets across different sensory modalities.Methods:Fifteen adults with normal,or corrected to normal,vision made saccades to either visual,auditory,tactile or proprioceptive targets.In the auditory condition a speaker was positioned at one of eight locations along a circle surrounding a central fixation point.In the proprioceptive condition the participant’s finger was placed at one of the eight locations.In the tactile condition participants were touched on their right forearm in one of four eccentric location,left and right of a central point.Eye movements were made in complete darkness.Results:We compared the precision and accuracy of the eye movements to tactile,proprioceptive,and auditory targets in the dark.Overall,both precision and accuracy of movements to non-visual targets were significantly lower compared to visual targets.Conclusions:These differences emphasize the central role of the visual system in saccade planning.
