Water covers most of the Earth’s surface and is nowhere near a good ecological or recreational state in many areas of the world.Moreover,only a small fraction of the water is potable.As climate change-induced extreme...Water covers most of the Earth’s surface and is nowhere near a good ecological or recreational state in many areas of the world.Moreover,only a small fraction of the water is potable.As climate change-induced extreme weather events become ever more prevalent,more and more issues arise,such as worsening water quality problems.Therefore,protecting invaluable and useable drinking water is critical.Environmental agencies must continuously check water sources to determine whether they are in a good or healthy state regarding pollutant levels and ecological status.The currently available tools are better suited for stationary laboratory use,and domain specialists lack suitable tools for onsite visualisation and interactive exploration of environmental data.Meanwhile,data collection for laboratory analysis requires substantial time and significant effort.We,therefore,developed an augmented reality system with a Microsoft HoloLens 2 device to explore the visualisation of water quality and status in situ.The developed prototype visualises geo-referenced sensor measurements incorporated into the perspective of the surroundings.Any users interested in water bodies’conditions can quickly examine and retrieve an overview of water body status using augmented reality and then take necessary steps to address the current situation.展开更多
Mixed reality technology has been increasingly used for navigation.While most MR-based navigation systems are currently based on hand-held devices,for example,smartphones,headmounted MR devices have become more and mo...Mixed reality technology has been increasingly used for navigation.While most MR-based navigation systems are currently based on hand-held devices,for example,smartphones,headmounted MR devices have become more and more popular in navigation.Much research has been conducted to investigate the navigation experience in MR.However,it is still unclear how ordinary users react to the first-person view and FOV(field of view)-limited navigation experience,especially in terms of spatial learning.In our study,we investigate how visualization in MR navigation affects spatial learning.More specifically,we test two related hypotheses:incorrect virtual information can lead users into incorrect spatial learning,and the visualization style of direction can influence users’spatial learning and experience.We designed a user interface in Microsoft HoloLens 2 and conducted a user study with 40 participants.The user study consists of a walking session in which users wear Microsoft HoloLens 2 to navigate to an unknown destination,pre-and post-walking questionnaires,sketch map drawing,and a semi-structured interview about the user interface design.The results provide preliminary confirmation that users’spatial learning can be misled by incorrect information,even in a small study area,but this misleading effect can be compensated by considerate visualization,for example,including lines instead of using only arrows as direction indicators.Arrows with or without lines as two visualization alternatives also influenced the user’s spatial learning and evaluation of the designed elements.Besides,the study shows that users’preferences for navigation interfaces are diverse,and an adaptable interface should be provided.The results contribute to the design of head-mounted MR-based navigation interfaces and the application of MR in navigation in general.展开更多
Virtual reality(VR) training technology in the mining industry is a new field of research and utilization.The successful application of VR training system is critical to mine safety and production. Through the statist...Virtual reality(VR) training technology in the mining industry is a new field of research and utilization.The successful application of VR training system is critical to mine safety and production. Through the statistics of the current research and applications of VR training systems in mining industry, all the input/output devices are classified. Based on the classifications of the input/output devices that are used in the VR system, the current VR training systems for the mining industry could be divided into three types: screen-based general type, projector-based customized type, and head-mounted display(HMD)-based intuitive type. By employing a VR headset, a smartphone and a leap motion device, an HMDbased intuitive type VR training system prototype for drilling in underground mines has been developed.Ten trainees tried both the HMD-based intuitive system and the screen-based general control system to compare the experiences and training effects. The results show that the HMD-based system can give a much better user experience and is easy to use. Three of the five components of a VR training system,namely, the user, the tasks, and software and database should be given more attention in future research.With more available technologies of input and output devices, VR engines, and system software, the VR training system will eventually yield much better training results, and will play a more important role in as a training tool for mine safety.展开更多
A new type of light fiehl display is proposed using a head-mounted display （HMD） and a micro structure array （MSA, lens array or pinhole array）. Each rendering point emits abundant rays from different directions i...A new type of light fiehl display is proposed using a head-mounted display （HMD） and a micro structure array （MSA, lens array or pinhole array）. Each rendering point emits abundant rays from different directions into the viewer＇s pupil, and at one time the dense light field is generated inside the exit pupil of the HMD through the eyepiece. Therefore, the proposed method not only solves the problem of accommodation and convergence conflict in a traditional HMD, but also drastically reduces the huge data in real three-dimensional （3D） display. To demonstrate the proposed method, a prototype is developed, which is capable of giving the observer a real perception of depth.展开更多
The efficiency balance phenomenon for see-through head-mounted displays with different microstructure con- ditions can be found both theoretically and using optical simulation software. A simple mathematical calculati...The efficiency balance phenomenon for see-through head-mounted displays with different microstructure con- ditions can be found both theoretically and using optical simulation software. A simple mathematical calculation is used to determine the relationship between the real image （see-through function） energy and the virtual image energy. The simulation is based on factors taken from previous research studies. It is found that the balance value of the optical efficiency remains almost constant （66.63% to 67.38%） under different microstructure conditions. In addition, suitable conditions for the microstructures in see-through head-mounted displays for daily applications can be predicted.展开更多
Measuring eye movement is a fundamental approach in cognitive science as it provides a variety of insightful parameters that reflect brain states such as visual attention and emotions.Combining eye-tracking with multi...Measuring eye movement is a fundamental approach in cognitive science as it provides a variety of insightful parameters that reflect brain states such as visual attention and emotions.Combining eye-tracking with multimodal neural recordings or manipulation techniques is beneficial for understanding the neural substrates of cognitive function.Many commercially-available and custom-built systems have been widely applied to awake,head-fixed small animals.However,the existing eyetracking systems used in freely-moving animals are still limited in terms of their compatibility with other devices and of the algorithm used to detect eye movements.Here,we report a novel system that integrates a general-purpose,easily compatible eye-tracking hardware with a robust eye feature-detection algorithm.With ultra-light hardware and a detachable design,the system allows for more implants to be added to the animal's exposed head and has a precise synchronization module to coordinate with other neural implants.Moreover,we systematically compared the performance of existing commonly-used pupil-detection approaches,and demonstrated that the proposed adaptive pupil feature-detection algorithm allows the analysis of more complex and dynamic eye-tracking data in freemoving animals.Synchronized eye-tracking and electroencephalogram recordings,as well as algorithm validation under five noise conditions,suggested that our system is flexibly adaptable and can be combined with a wide range of neural manipulation and recording technologies.展开更多
Spatially resolved transcriptomics is an emerging class of high-throughput technologies that enable biologists to systematically investigate the expression of genes along with spatial information.Upon data acquisition...Spatially resolved transcriptomics is an emerging class of high-throughput technologies that enable biologists to systematically investigate the expression of genes along with spatial information.Upon data acquisition,one major hurdle is the subsequent interpretation and visualization of the datasets acquired.To address this challenge,VR-Cardiomics is presented,which is a novel data visualization system with interactive functionalities designed to help biologists interpret spatially resolved transcriptomic datasets.By implementing the system in two separate immersive environments,fish tank virtual reality(FTVR)and head-mounted display virtual reality(HMD-VR),biologists can interact with the data in novel ways not previously possible,such as visually exploring the gene expression patterns of an organ,and comparing genes based on their 3D expression profiles.Further,a biologist-driven use-case is presented,in which immersive environments facilitate biologists to explore and compare the heart expression profiles of different genes.展开更多
To improve and develop education systems,the communication between instructors and learners in a class during the learning process is of utmost importance.Currently the presentations of 3D models using mixed reality(M...To improve and develop education systems,the communication between instructors and learners in a class during the learning process is of utmost importance.Currently the presentations of 3D models using mixed reality(MR)technology can be used to avoid misinterpretations of oral and 2D model presentations.As an independent concept and MR applications,MR combines the excellent of each virtual reality(VR)and augmented reality(AR).This work aims to present the descriptions of MR systems,which include its devices,applications,and literature reviews and proposes computer vision tracking using the AR Toolkit Tracking Library.The focus of this work will be on creating 3D models and implementing in Unity 3D using the Vuforia SDK platform to develop VR and AR applications for architectural presentations.展开更多
This review considers the modern industrial applications of augmented reality headsets.It draws upon a synthesis of information from open sources and press releases of companies,as well as the first-hand experiences o...This review considers the modern industrial applications of augmented reality headsets.It draws upon a synthesis of information from open sources and press releases of companies,as well as the first-hand experiences of industry representatives.Furthermore,the research incorporates insights from both profile events and in-depth discussions with skilled professionals.A specific focus is placed on the ergonomic characteristics of headsets:image quality,user-friendliness,etc.To provide an objective evaluation of the various headsets,a metric has been proposed which is dependent on the specific application case.This enables a comprehensive comparison of the various devices in terms of their quantitative characteristics,which is of particular importance for the formation of a rapidly developing industry.展开更多
As humans and robots work closer together than ever,anthropomorphic robotic arms with intuitive human-robot interaction interfaces have drawn massive attention to improving the quality of robot-assisted manipulation.I...As humans and robots work closer together than ever,anthropomorphic robotic arms with intuitive human-robot interaction interfaces have drawn massive attention to improving the quality of robot-assisted manipulation.In pursuit of this,we designed a dedicated 7-degrees-of-freedom(DoF)anthropomorphic robotic arm having three compact differential joints and a head-mounted gaze tracker enabling head-pose-tracked 3D gaze estimation.Moreover,two key challenges were addressed to achieve accurate robot-assisted manipulation of the object indicated by the direction of human gaze.First,a novel predictive pupil feature was proposed for 3D gaze estimation.Differing from most existing features subjected to the common paraxial approximation assumption,the proposed novel predictive pupil feature considered the light refraction at two corneal surfaces with a more realistic eye model,significantly improving the 3D gaze estimation accuracy when the eyeball rotates at large angles.Second,a novel optimization-based approach was developed to efficiently compensate for the posture errors of the designed 7-DoF anthropomorphic robotic arm for accurate manipulation.Compared with the existing Jacobian-based or optimization-based approaches with nominal joint values as iteration initial,the proposed approach computed the optimal iteration initial and realized faster convergence for real-time posture error compensation.With the posture error compensation in real time and 3D gaze estimated accurately,the human can command accurate robot-assisted manipulation using his eyes intuitively.The proposed system was successfully tested on five healthy subjects.展开更多
基金supported by the Freshwater Competence Centre,Academy of Finland(Decision No.345008)the Nordic University Cooperation on Edge Intelligence(Grant No.168043).
文摘Water covers most of the Earth’s surface and is nowhere near a good ecological or recreational state in many areas of the world.Moreover,only a small fraction of the water is potable.As climate change-induced extreme weather events become ever more prevalent,more and more issues arise,such as worsening water quality problems.Therefore,protecting invaluable and useable drinking water is critical.Environmental agencies must continuously check water sources to determine whether they are in a good or healthy state regarding pollutant levels and ecological status.The currently available tools are better suited for stationary laboratory use,and domain specialists lack suitable tools for onsite visualisation and interactive exploration of environmental data.Meanwhile,data collection for laboratory analysis requires substantial time and significant effort.We,therefore,developed an augmented reality system with a Microsoft HoloLens 2 device to explore the visualisation of water quality and status in situ.The developed prototype visualises geo-referenced sensor measurements incorporated into the perspective of the surroundings.Any users interested in water bodies’conditions can quickly examine and retrieve an overview of water body status using augmented reality and then take necessary steps to address the current situation.
基金supported by the China Scholarship Council[Grant No.201806040219 and Grant No.202006040025]。
文摘Mixed reality technology has been increasingly used for navigation.While most MR-based navigation systems are currently based on hand-held devices,for example,smartphones,headmounted MR devices have become more and more popular in navigation.Much research has been conducted to investigate the navigation experience in MR.However,it is still unclear how ordinary users react to the first-person view and FOV(field of view)-limited navigation experience,especially in terms of spatial learning.In our study,we investigate how visualization in MR navigation affects spatial learning.More specifically,we test two related hypotheses:incorrect virtual information can lead users into incorrect spatial learning,and the visualization style of direction can influence users’spatial learning and experience.We designed a user interface in Microsoft HoloLens 2 and conducted a user study with 40 participants.The user study consists of a walking session in which users wear Microsoft HoloLens 2 to navigate to an unknown destination,pre-and post-walking questionnaires,sketch map drawing,and a semi-structured interview about the user interface design.The results provide preliminary confirmation that users’spatial learning can be misled by incorrect information,even in a small study area,but this misleading effect can be compensated by considerate visualization,for example,including lines instead of using only arrows as direction indicators.Arrows with or without lines as two visualization alternatives also influenced the user’s spatial learning and evaluation of the designed elements.Besides,the study shows that users’preferences for navigation interfaces are diverse,and an adaptable interface should be provided.The results contribute to the design of head-mounted MR-based navigation interfaces and the application of MR in navigation in general.
基金funded by the ‘‘twelfth five” National Key Technology Research and Development Program of the Ministry of Science and Technology of China (Grant No. 2015BAK10B00)
文摘Virtual reality(VR) training technology in the mining industry is a new field of research and utilization.The successful application of VR training system is critical to mine safety and production. Through the statistics of the current research and applications of VR training systems in mining industry, all the input/output devices are classified. Based on the classifications of the input/output devices that are used in the VR system, the current VR training systems for the mining industry could be divided into three types: screen-based general type, projector-based customized type, and head-mounted display(HMD)-based intuitive type. By employing a VR headset, a smartphone and a leap motion device, an HMDbased intuitive type VR training system prototype for drilling in underground mines has been developed.Ten trainees tried both the HMD-based intuitive system and the screen-based general control system to compare the experiences and training effects. The results show that the HMD-based system can give a much better user experience and is easy to use. Three of the five components of a VR training system,namely, the user, the tasks, and software and database should be given more attention in future research.With more available technologies of input and output devices, VR engines, and system software, the VR training system will eventually yield much better training results, and will play a more important role in as a training tool for mine safety.
基金partially supported by the National Basic Research Program of China(No.2013CB328805)the National Science Foundation of China(NSFC,No.61205024,61178038)the National Key Technology R&D Program(No.2012BAH64F03)
文摘A new type of light fiehl display is proposed using a head-mounted display （HMD） and a micro structure array （MSA, lens array or pinhole array）. Each rendering point emits abundant rays from different directions into the viewer＇s pupil, and at one time the dense light field is generated inside the exit pupil of the HMD through the eyepiece. Therefore, the proposed method not only solves the problem of accommodation and convergence conflict in a traditional HMD, but also drastically reduces the huge data in real three-dimensional （3D） display. To demonstrate the proposed method, a prototype is developed, which is capable of giving the observer a real perception of depth.
基金supported in part by the Ministry of Science and Technology,Taiwan,project number MOST104-2220-E-009-006in part by the "Aim for the Top University Plan" of the National Chiao Tung University and the Ministry of Education,Taiwan,China
文摘The efficiency balance phenomenon for see-through head-mounted displays with different microstructure con- ditions can be found both theoretically and using optical simulation software. A simple mathematical calculation is used to determine the relationship between the real image （see-through function） energy and the virtual image energy. The simulation is based on factors taken from previous research studies. It is found that the balance value of the optical efficiency remains almost constant （66.63% to 67.38%） under different microstructure conditions. In addition, suitable conditions for the microstructures in see-through head-mounted displays for daily applications can be predicted.
基金supported in part by the National Key R&D Program of China(2021ZD0203902 and 2018YFA0701403)the Key Area R&D Program of Guangdong Province(2018B030338001 and 2018B030331001)+9 种基金the National Natural Science Foundation of China(31500861,31630031,91732304,and 31930047)the Chang Jiang Scholars Program and the Ten Thousand Talent Program,the International Big Science Program Cultivating Project of the Chinese Academy of Science(CAS)(172644KYS820170004)the Strategic Priority Research Program of the CAS(XDB32030100)the Youth Innovation Promo-tion Association of the CAS(2017413)Shenzhen Government Basic Research Grants(JCYJ20170411140807570,JCYJ20170413164535041)the Science,Technology and Innovation Commission of Shenzhen Municipality(JCYJ20160429185235132)a Helmholtz-CAS joint research grant(GJHZ1508)the Guangdong Provincial Key Laboratory of Brain Connectome and Behavior(2017B030301017)the Guangdong Special Support Program,the Key Laboratory of the CAS(2019DP173024)the Shenzhen Key Science and Technology Infrastructure Planning Project(ZDKJ20190204002)。
文摘Measuring eye movement is a fundamental approach in cognitive science as it provides a variety of insightful parameters that reflect brain states such as visual attention and emotions.Combining eye-tracking with multimodal neural recordings or manipulation techniques is beneficial for understanding the neural substrates of cognitive function.Many commercially-available and custom-built systems have been widely applied to awake,head-fixed small animals.However,the existing eyetracking systems used in freely-moving animals are still limited in terms of their compatibility with other devices and of the algorithm used to detect eye movements.Here,we report a novel system that integrates a general-purpose,easily compatible eye-tracking hardware with a robust eye feature-detection algorithm.With ultra-light hardware and a detachable design,the system allows for more implants to be added to the animal's exposed head and has a precise synchronization module to coordinate with other neural implants.Moreover,we systematically compared the performance of existing commonly-used pupil-detection approaches,and demonstrated that the proposed adaptive pupil feature-detection algorithm allows the analysis of more complex and dynamic eye-tracking data in freemoving animals.Synchronized eye-tracking and electroencephalogram recordings,as well as algorithm validation under five noise conditions,suggested that our system is flexibly adaptable and can be combined with a wide range of neural manipulation and recording technologies.
基金This project was partly funded by the Deutsche Forschungsgemeinschaft(DFG,German Research Foundation)-Project-ID 251654672-TRR 161by DFG Center of Excellence 2117“Centre for the Advanced Study of Collective Behaviour”(ID 422037984)M.R.was funded by an NH&MRC/Heart Foundation Career Development Fellowship and by an Australian Research Council Discovery Project DP190102771 Grant.
文摘Spatially resolved transcriptomics is an emerging class of high-throughput technologies that enable biologists to systematically investigate the expression of genes along with spatial information.Upon data acquisition,one major hurdle is the subsequent interpretation and visualization of the datasets acquired.To address this challenge,VR-Cardiomics is presented,which is a novel data visualization system with interactive functionalities designed to help biologists interpret spatially resolved transcriptomic datasets.By implementing the system in two separate immersive environments,fish tank virtual reality(FTVR)and head-mounted display virtual reality(HMD-VR),biologists can interact with the data in novel ways not previously possible,such as visually exploring the gene expression patterns of an organ,and comparing genes based on their 3D expression profiles.Further,a biologist-driven use-case is presented,in which immersive environments facilitate biologists to explore and compare the heart expression profiles of different genes.
文摘To improve and develop education systems,the communication between instructors and learners in a class during the learning process is of utmost importance.Currently the presentations of 3D models using mixed reality(MR)technology can be used to avoid misinterpretations of oral and 2D model presentations.As an independent concept and MR applications,MR combines the excellent of each virtual reality(VR)and augmented reality(AR).This work aims to present the descriptions of MR systems,which include its devices,applications,and literature reviews and proposes computer vision tracking using the AR Toolkit Tracking Library.The focus of this work will be on creating 3D models and implementing in Unity 3D using the Vuforia SDK platform to develop VR and AR applications for architectural presentations.
基金support of“Priority 2030”program at the Bauman Moscow State Technical University.O.L.A.and M.V.S.acknowledge the financial support of the Ministry of Science and Higher Education of the Russian Federation grant(Agreement dated 06.03.2024 number 075-02-2024-1519)for the experimental research,carried out using the infrastructure of the Educational Design Center for Opto-and Microelectronics of the Bauman Moscow State Technical Universitysupport of the Ministry of Science and Higher Education of the Russian Federation(Passport No.2019-0903).
文摘This review considers the modern industrial applications of augmented reality headsets.It draws upon a synthesis of information from open sources and press releases of companies,as well as the first-hand experiences of industry representatives.Furthermore,the research incorporates insights from both profile events and in-depth discussions with skilled professionals.A specific focus is placed on the ergonomic characteristics of headsets:image quality,user-friendliness,etc.To provide an objective evaluation of the various headsets,a metric has been proposed which is dependent on the specific application case.This enables a comprehensive comparison of the various devices in terms of their quantitative characteristics,which is of particular importance for the formation of a rapidly developing industry.
基金supported by the National Natural Science Foundation of China(Grant Nos.52027806,52435005,92248304,52075191).
文摘As humans and robots work closer together than ever,anthropomorphic robotic arms with intuitive human-robot interaction interfaces have drawn massive attention to improving the quality of robot-assisted manipulation.In pursuit of this,we designed a dedicated 7-degrees-of-freedom(DoF)anthropomorphic robotic arm having three compact differential joints and a head-mounted gaze tracker enabling head-pose-tracked 3D gaze estimation.Moreover,two key challenges were addressed to achieve accurate robot-assisted manipulation of the object indicated by the direction of human gaze.First,a novel predictive pupil feature was proposed for 3D gaze estimation.Differing from most existing features subjected to the common paraxial approximation assumption,the proposed novel predictive pupil feature considered the light refraction at two corneal surfaces with a more realistic eye model,significantly improving the 3D gaze estimation accuracy when the eyeball rotates at large angles.Second,a novel optimization-based approach was developed to efficiently compensate for the posture errors of the designed 7-DoF anthropomorphic robotic arm for accurate manipulation.Compared with the existing Jacobian-based or optimization-based approaches with nominal joint values as iteration initial,the proposed approach computed the optimal iteration initial and realized faster convergence for real-time posture error compensation.With the posture error compensation in real time and 3D gaze estimated accurately,the human can command accurate robot-assisted manipulation using his eyes intuitively.The proposed system was successfully tested on five healthy subjects.
