Surgical navigation has evolved significantly through advances in augmented reality,virtual reality,and mixed reality,improving precision and safety across many clinical applications,including neurosurgery,maxillofaci...Surgical navigation has evolved significantly through advances in augmented reality,virtual reality,and mixed reality,improving precision and safety across many clinical applications,including neurosurgery,maxillofacial,spinal,and arthroplasty procedures.By integrating preoperative imaging with real-time intraoperative data,these systems provide dynamic guidance,reduce radiation exposure,and minimize tissue damage.Key challenges persist,including intraoperative registration accuracy,flexible tissue deformation,respiratory compensation,and real-time imaging quality.Emerging solutions include artificial intelligence-driven segmentation,deformation-field modeling,and hybrid registration techniques.Future developments will include lightweight,portable systems,improved non-rigid registration algorithms,and greater clinical adoption.Despite advances in rigid-tissue applications,soft-tissue navigation requires additional innovation to address motion variability and registration reliability,ultimately advancing minimally invasive surgery and precision medicine.展开更多
Augmented reality(AR)is an emerging dynamic technology that effectively supports education across different levels.The increased use of mobile devices has an even greater impact.As the demand for AR applications in ed...Augmented reality(AR)is an emerging dynamic technology that effectively supports education across different levels.The increased use of mobile devices has an even greater impact.As the demand for AR applications in education continues to increase,educators actively seek innovative and immersive methods to engage students in learning.However,exploring these possibilities also entails identifying and overcoming existing barriers to optimal educational integration.Concurrently,this surge in demand has prompted the identification of specific barriers,one of which is three-dimensional(3D)modeling.Creating 3D objects for augmented reality education applications can be challenging and time-consuming for the educators.To address this,we have developed a pipeline that creates realistic 3D objects from the two-dimensional(2D)photograph.Applications for augmented and virtual reality can then utilize these created 3D objects.We evaluated the proposed pipeline based on the usability of the 3D object and performance metrics.Quantitatively,with 117 respondents,the co-creation team was surveyed with openended questions to evaluate the precision of the 3D object created by the proposed photogrammetry pipeline.We analyzed the survey data using descriptive-analytical methods and found that the proposed pipeline produces 3D models that are positively accurate when compared to real-world objects,with an average mean score above 8.This study adds new knowledge in creating 3D objects for augmented reality applications by using the photogrammetry technique;finally,it discusses potential problems and future research directions for 3D objects in the education sector.展开更多
Endoscopic transnasal optic nerve decompression surgery plays a crucial role in minimal invasive treatment of complex traumatic optic neuropathy.However,a major challenge faced during the procedure is the inability to...Endoscopic transnasal optic nerve decompression surgery plays a crucial role in minimal invasive treatment of complex traumatic optic neuropathy.However,a major challenge faced during the procedure is the inability to visualize the optic nerve intraoperatively.To address this issue,an endoscopic image-based augmented reality surgical navigation system is developed in this study.The system aims to virtually fuse the optic nerve onto the endoscopic images,assisting surgeons in determining the optic nerve’s position and reducing surgical risks.First,a calibration algorithm based on a checkerboard grid of immobile points is proposed,building upon existing calibration methods.Additionally,to tackle accuracy issues associated with augmented reality technology,an optical navigation and visual fusion compensation algorithm is proposed to improve the intraoperative tracking accuracy.To evaluate the system’s performance,model experiments were meticulously designed and conducted.The results confirm the accuracy and stability of the proposed system,with an average tracking error of(0.99±0.46)mm.This outcome demonstrates the effectiveness of the proposed algorithm in improving the augmented reality surgical navigation system’s accuracy.Furthermore,the system successfully displays hidden optic nerves and other deep tissues,thus showcasing the promising potential for future applications in orbital and maxillofacial surgery.展开更多
Augmented reality(AR)is a technology that superimposes digital information onto real-world objects via head-mounted display devices to improve surgical finesse through visually enhanced medical information.With the ra...Augmented reality(AR)is a technology that superimposes digital information onto real-world objects via head-mounted display devices to improve surgical finesse through visually enhanced medical information.With the rapid development of digital technology,AR has been increasingly adopted in orthopedic surgeries across the globe,especially in total knee arthroplasty procedures which demand high precision.By overlaying digital information onto the surgeon's field of view,AR systems enhance precision,improve alignment accuracy,and reduce the risk of complications associated with malalignment.Some concerns have been raised despite accuracy,including the learning curve,long-term outcomes,and technical limitations.Furthermore,it is essential for health practitioners to gain trust in the utilisation of AR.展开更多
The aim of this study was to assess the potential of surgical guides as a complementary tool to augmented reality(AR)in enhancing the safety and precision of pedicle screw placement in spinal surgery.Four trainers wer...The aim of this study was to assess the potential of surgical guides as a complementary tool to augmented reality(AR)in enhancing the safety and precision of pedicle screw placement in spinal surgery.Four trainers were divided into the AR navigation group using surgical guides and the free-hand group.Each group consisted of a novice and an experienced spine surgeon.A total of 80 pedicle screws were implanted.First,the AR group reconstructed the 3D model and planned the screw insertion route according to the computed tomography data of L2 lumbar vertebrae.Then,the Microsoft HoloLens™2 was used to identify the vertebral model,and the planned virtual path was superimposed on the real cone model.Next,the screw was placed according to the projected trajectory.Finally,Micron Tracker was used to measure the deviation of screws from the preoperatively planned trajectory,and pedicle screws were evaluated using the Gertzbein-Robbins scale.In the AR group,the linear deviations of the experienced doctor and the novice were(1.59±0.39)mm and(1.73±0.52)mm respectively,and the angle deviations were 2.72°±0.61°and 2.87°±0.63°respectively.In the free-hand group,the linear deviations of the experienced doctor and the novice were(2.88±0.58)mm and(5.25±0.62)mm respectively,and the angle deviations were 4.41°±1.18°and 7.15°±1.45°respectively.Both kinds of deviations between the two groups were significantly different(P<0.05).The screw accuracy rate was 95%in the AR navigation group and 77.5%in the free-hand group.The results of this study indicate that the integration of surgical guides and AR is an innovative technique that can substantially enhance the safety and precision of spinal surgery and assist inexperienced doctors in completing the surgery.展开更多
The purpose of this study is to establish a multivariate nonlinear regression mathematical model to predict the displacement of tumor during brain tumor resection surgery.And the study will be integrated with augmente...The purpose of this study is to establish a multivariate nonlinear regression mathematical model to predict the displacement of tumor during brain tumor resection surgery.And the study will be integrated with augmented reality technology to achieve three-dimensional visualization,thereby enhancing the complete resection rate of tumor and the success rate of surgery.Based on the preoperative MRI data of the patients,a 3D virtual model is reconstructed and 3D printed.A brain biomimetic model is created using gel injection molding.By considering cerebrospinal fluid loss and tumor cyst fluid loss as independent variables,the highest point displacement in the vertical bone window direction is determined as the dependent variable after positioning the patient for surgery.An orthogonal experiment is conducted on the biomimetic model to establish a predictive model,and this model is incorporated into the augmented reality navigation system.To validate the predictive model,five participants wore HoloLens2 devices,overlaying the patient’s 3D virtual model onto the physical head model.Subsequently,the spatial coordinates of the tumor’s highest point after displacement were measured on both the physical and virtual models(actual coordinates and predicted coordinates,respectively).The difference between these coordinates represents the model’s prediction error.The results indicate that the measured and predicted errors for the displacement of the tumor’s highest point on the X and Y axes range from−0.6787 mm to 0.2957 mm and−0.4314 mm to 0.2253 mm,respectively.The relative errors for each experimental group are within 10%,demonstrating a good fit of the model.This method of establishing a regression model represents a preliminary attempt to predict brain tumor displacement in specific situations.It also provides a new approach for surgeons.By combining augmented reality visualization,it addresses the need for predicting tumor displacement and precisely locating brain anatomical structures in a simple and cost-effective manner.展开更多
With the iteration and upgrading of medical technology and the continuous growth of public health demands,the quality of nursing services has become a core indicator for measuring the effectiveness of the medical syst...With the iteration and upgrading of medical technology and the continuous growth of public health demands,the quality of nursing services has become a core indicator for measuring the effectiveness of the medical system.The clinical practice ability of nursing staff is directly related to the safety of patient diagnosis and treatment and the rehabilitation process.However,the current clinical nursing talent training model is facing bottlenecks such as limited practical scenarios and fragmented case cognition.This study focuses on the teaching application of augmented reality(AR)technology in hospital Settings and systematically reviews the research progress on the improvement of clinical practice ability of trainee nurses based on the AR immersive teaching model.By constructing a clinical teaching scenario that integrates virtual and real,AR technology can dynamically simulate complex case handling processes and enhance nursing students’three-dimensional cognition of condition assessment,operation norms,and emergency plans.Hospitals,as the core base for practical teaching,can effectively shorten the connection cycle between theoretical teaching and clinical practice by integrating AR technology,improve the clinical practice level of trainee nurses,and provide an innovative model for optimizing the path of clinical nursing talent cultivation.展开更多
Closed thoracic drainage can be performed using a steel-needle-guided chest tube to treat pleural effusion or pneumothorax in clinics.However,the puncture procedure during surgery is invisible,increasing the risk of s...Closed thoracic drainage can be performed using a steel-needle-guided chest tube to treat pleural effusion or pneumothorax in clinics.However,the puncture procedure during surgery is invisible,increasing the risk of surgical failure.Therefore,it is necessary to design a visualization system for closed thoracic drainage.Augmented reality(AR)technology can assist in visualizing the internal anatomical structure and determining the insertion point on the body surface.The structure of the currently used steel-needle-guided chest tube was modified by integrating it with an ultrafine diameter camera to provide real-time visualization of the puncture process.After simulation experiments,the overall registration error of the AR method was measured to be within(3.59±0.53)mm,indicating its potential for clinical application.The ultrafine diameter camera module and improved steel-needle-guided chest tube can timely reflect the position of the needle tip in the human body.A comparative experiment showed that video guidance could improve the safety of the puncture process compared to the traditional method.Finally,a qualitative evaluation of the usability of the system was conducted through a questionnaire.This system facilitates the visualization of closed thoracic drainage puncture procedure and pro-vides an implementation scheme to enhance the accuracy and safety of the operative step,which is conducive to reducing the learning curve and improving the proficiency of the doctors.展开更多
Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the v...Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.Data Sources: The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the Pub Med database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.Results: In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery,which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.Conclusions: With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling,and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.展开更多
The popularity of wearable devices and smartphones has fueled the development of Mobile Augmented Reality(MAR),which provides immersive experiences over the real world using techniques,such as computer vision and deep...The popularity of wearable devices and smartphones has fueled the development of Mobile Augmented Reality(MAR),which provides immersive experiences over the real world using techniques,such as computer vision and deep learning.However,the hardware-specific MAR is costly and heavy,and the App-based MAR requires an additional download and installation and it also lacks cross-platform ability.These limitations hamper the pervasive promotion of MAR.This paper argues that mobile Web AR(MWAR)holds the potential to become a practical and pervasive solution that can effectively scale to millions of end-users because MWAR can be developed as a lightweight,cross-platform,and low-cost solution for end-to-end delivery of MAR.The main challenges for making MWAR a reality lie in the low efficiency for dense computing in Web browsers,a large delay for real-time interactions over mobile networks,and the lack of standardization.The good news is that the newly emerging 5G and Beyond 5G(B5G)cellular networks can mitigate these issues to some extent via techniques such as network slicing,device-to-device communication,and mobile edge computing.In this paper,we first give an overview of the challenges and opportunities of MWAR in the 5G era.Then we describe our design and development of a generic service-oriented framework(called MWAR5)to provide a scalable,flexible,and easy to deploy MWAR solution.We evaluate the performance of our MWAR5 system in an actually deployed 5G trial network under the collaborative configurations,which shows encouraging results.Moreover,we also share the experiences and insights from our development and deployment,including some exciting future directions of MWAR over 5G and B5G networks.展开更多
To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. T...To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. Three-dimensional surface models of the jawbones, based on the computed tomography data, were used to create the integral videography images of a subject＇s maxillofacial area. The three-dimensional augmented reality system （integral videography display, computed tomography, a position tracker and a computer） was used to generate a three-dimensional overlay that was projected on the surgical site via a half-silvered mirror. Thereafter, a feasibility study was performed on a volunteer. The accuracy of this system was verified on a solid model while simulating bone resection. Positional registration was attained by identifying and tracking the patient/surgical instrument＇s position. Thus, integral videography images of jawbones, teeth and the surgical tool were superimposed in the correct position. Stereoscopic images viewed from various angles were accurately displayed. Change in the viewing angle did not negatively affect the surgeon＇s ability to simultaneously observe the three-dimensional images and the patient, without special glasses. The difference in three-dimensional position of each measuring point on the solid model and augmented reality navigation was almost negligible （〈1 mm）; this indicates that the system was highly accurate. This augmented reality system was highly accurate and effective for surgical navigation and for overlaying a three-dimensional computed tomography image on a patient＇s surgical area, enabling the surgeon to understand the positional relationship between the preoperative image and the actual surgical site, with the naked eye.展开更多
Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass optical head-mounted display to train urology reside...Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass optical head-mounted display to train urology residents in how to place an inflatable penile prosthesis. We use the phrase Augmented Reality Assisted Surgery to describe this novel application of augmented reality in the setting of surgery. The application demonstrates the steps of the surgical procedure of inflatable penile prosthesis placement. It also contains software that allows for detection of interest points using a camera feed from the optical head-mounted display to enable faculty to interact with residents during placement of the penile prosthesis. Urology trainees and faculty who volunteered to take part in the study were given time to experience the technology in the operative or perioperative setting and asked to complete a feedback survey. From 30 total participants using a lO-point scale, educational usefulness was rated 8.6, ease of navigation was rated 7.6, likelihood to use was rated 7.4, and distraction in operating room was rated 4.9. When stratified between trainees and faculty, trainees found the technology more educationally useful, and less distracting. Overall, 81% of the participants want this technology in their residency program, and 93% see this technology in the operating room in the future. Further development of this technology is warranted before full release, and further studies are necessary to better characterize the effectiveness of Augmented Reality Assisted Surgery in urologic surgical training.展开更多
The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery s...The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.展开更多
Augmented reality(AR)and virtual reality(VR)are two novel display technologies that are under updates.The essential feature of AR/VR is the full-color display that requires high pixel densities.To generate three-color...Augmented reality(AR)and virtual reality(VR)are two novel display technologies that are under updates.The essential feature of AR/VR is the full-color display that requires high pixel densities.To generate three-color pixels,the fluorescent color conversion layer inevitably includes green and red pixels.To fabricate such sort of display kits,inkjet printing is a promising way to position the color conversion layers.In this review article,the progress of AR/VR technologies is first reviewed,and in succession,the state of the art of inkjet printing,as well as two key issues-the optimization of ink and the reduction of coffee-ring effects,are introduced.Finally,some potential problems associated with the color converting layer are highlighted.展开更多
Tracking registration is a key issue in augmented reality applications,particularly where there are no artificial identifier placed manually.In this paper,an efficient markerless tracking registration algorithm which ...Tracking registration is a key issue in augmented reality applications,particularly where there are no artificial identifier placed manually.In this paper,an efficient markerless tracking registration algorithm which combines the detector and the tracker is presented for the augmented reality system.We capture the target images in real scenes as template images,use the random ferns classifier for target detection and solve the problem of reinitialization after tracking registration failures due to changes in ambient lighting or occlusion of targets.Once the target has been successfully detected,the pyramid Lucas-Kanade(LK)optical flow tracker is used to track the detected target in real time to solve the problem of slow speed.The least median of squares(LMedS)method is used to adaptively calculate the homography matrix,and then the three-dimensional pose is estimated and the virtual object is rendered and registered.Experimental results demonstrate that the algorithm is more accurate,faster and more robust.展开更多
In order to overcome the defects where the surface of the object lacks sufficient texture features and the algorithm cannot meet the real-time requirements of augmented reality,a markerless augmented reality tracking ...In order to overcome the defects where the surface of the object lacks sufficient texture features and the algorithm cannot meet the real-time requirements of augmented reality,a markerless augmented reality tracking registration method based on multimodal template matching and point clouds is proposed.The method first adapts the linear parallel multi-modal LineMod template matching method with scale invariance to identify the texture-less target and obtain the reference image as the key frame that is most similar to the current perspective.Then,we can obtain the initial pose of the camera and solve the problem of re-initialization because of tracking registration interruption.A point cloud-based method is used to calculate the precise pose of the camera in real time.In order to solve the problem that the traditional iterative closest point(ICP)algorithm cannot meet the real-time requirements of the system,Kdtree(k-dimensional tree)is used under the graphics processing unit(GPU)to replace the part of finding the nearest points in the original ICP algorithm to improve the speed of tracking registration.At the same time,the random sample consensus(RANSAC)algorithm is used to remove the error point pairs to improve the accuracy of the algorithm.The results show that the proposed tracking registration method has good real-time performance and robustness.展开更多
The massive technological advancements around the world have created significant challenging competition among companies where each of the companies tries to attract the customers using different techniques. One of th...The massive technological advancements around the world have created significant challenging competition among companies where each of the companies tries to attract the customers using different techniques. One of the recent tech- niques is Augmented Reality (AR). The AR is a new technology which is capable of presenting possibilities that are difficult for other technologies to offer and meet. Nowadays, numerous augmented reality applications have been used in the industry of different kinds and disseminated all over the world. AR will really alter the way individuals view the world. The AR is yet in its initial phases of research and development at different colleges and high-tech institutes. Throughout the last years, AR apps became transportable and generally available on various devices. Besides, AR be- gins to occupy its place in our audio-visual media and to be used in various fields in our life in tangible and exciting ways such as news, sports and is used in many domains in our life such as electronic commerce, promotion, design, and business. In addition, AR is used to facilitate the learning whereas it enables students to access location-specific infor- mation provided through various sources. Such growth and spread of AR applications pushes organizations to compete one another, every one of them exerts its best to gain the customers. This paper provides a comprehensive study of AR including its history, architecture, applications, current challenges and future trends.展开更多
Nonlinear errors always exist in data obtained from tracker in augmented reality （AR）, which badly influence the effect of AR. This paper proposes to rectify the errors using BP neural network. As BP neural network ...Nonlinear errors always exist in data obtained from tracker in augmented reality （AR）, which badly influence the effect of AR. This paper proposes to rectify the errors using BP neural network. As BP neural network is prone to getting into local extrema and convergence is slow, genetic algorithm is employed to optimize the initial weights and threshold of neural network. This paper discusses how to set the crucial parameters in the algorithm. Experimental results show that the method ensures that the neural network achieves global convergence quickly and correctly. Tracking precision of AR system is improved after the tracker is rectified, and the third dimension of AR system is enhanced.展开更多
基金Supported by the National Natural Science Foundation of China(NSFC)under Grants 62025104,62422102,62331005,62301034,and U22A2052the Beijing Natural Science Foundation-Daxing Innovation Joint Fund(L256040).
文摘Surgical navigation has evolved significantly through advances in augmented reality,virtual reality,and mixed reality,improving precision and safety across many clinical applications,including neurosurgery,maxillofacial,spinal,and arthroplasty procedures.By integrating preoperative imaging with real-time intraoperative data,these systems provide dynamic guidance,reduce radiation exposure,and minimize tissue damage.Key challenges persist,including intraoperative registration accuracy,flexible tissue deformation,respiratory compensation,and real-time imaging quality.Emerging solutions include artificial intelligence-driven segmentation,deformation-field modeling,and hybrid registration techniques.Future developments will include lightweight,portable systems,improved non-rigid registration algorithms,and greater clinical adoption.Despite advances in rigid-tissue applications,soft-tissue navigation requires additional innovation to address motion variability and registration reliability,ultimately advancing minimally invasive surgery and precision medicine.
文摘Augmented reality(AR)is an emerging dynamic technology that effectively supports education across different levels.The increased use of mobile devices has an even greater impact.As the demand for AR applications in education continues to increase,educators actively seek innovative and immersive methods to engage students in learning.However,exploring these possibilities also entails identifying and overcoming existing barriers to optimal educational integration.Concurrently,this surge in demand has prompted the identification of specific barriers,one of which is three-dimensional(3D)modeling.Creating 3D objects for augmented reality education applications can be challenging and time-consuming for the educators.To address this,we have developed a pipeline that creates realistic 3D objects from the two-dimensional(2D)photograph.Applications for augmented and virtual reality can then utilize these created 3D objects.We evaluated the proposed pipeline based on the usability of the 3D object and performance metrics.Quantitatively,with 117 respondents,the co-creation team was surveyed with openended questions to evaluate the precision of the 3D object created by the proposed photogrammetry pipeline.We analyzed the survey data using descriptive-analytical methods and found that the proposed pipeline produces 3D models that are positively accurate when compared to real-world objects,with an average mean score above 8.This study adds new knowledge in creating 3D objects for augmented reality applications by using the photogrammetry technique;finally,it discusses potential problems and future research directions for 3D objects in the education sector.
基金the National Natural Science Foundation of China(Nos.82330063 and M-0019)the Interdisciplinary Program of Shanghai Jiao Tong University(Nos.YG2022QN056,YG2023ZD19,and YG2023ZD15)+2 种基金the Cross Disciplinary Research Fund of Shanghai Ninth People’s Hospital,Shanghai Jiao Tong University School of Medicine(No.JYJC202115)the Translation Clinical R&D Project of Medical Robot of Shanghai Ninth People’s Hospital,Shanghai Jiao Tong University School of Medicine(No.IMR-NPH202002)the Shanghai Key Clinical Specialty,Shanghai Eye Disease Research Center(No.2022ZZ01003)。
文摘Endoscopic transnasal optic nerve decompression surgery plays a crucial role in minimal invasive treatment of complex traumatic optic neuropathy.However,a major challenge faced during the procedure is the inability to visualize the optic nerve intraoperatively.To address this issue,an endoscopic image-based augmented reality surgical navigation system is developed in this study.The system aims to virtually fuse the optic nerve onto the endoscopic images,assisting surgeons in determining the optic nerve’s position and reducing surgical risks.First,a calibration algorithm based on a checkerboard grid of immobile points is proposed,building upon existing calibration methods.Additionally,to tackle accuracy issues associated with augmented reality technology,an optical navigation and visual fusion compensation algorithm is proposed to improve the intraoperative tracking accuracy.To evaluate the system’s performance,model experiments were meticulously designed and conducted.The results confirm the accuracy and stability of the proposed system,with an average tracking error of(0.99±0.46)mm.This outcome demonstrates the effectiveness of the proposed algorithm in improving the augmented reality surgical navigation system’s accuracy.Furthermore,the system successfully displays hidden optic nerves and other deep tissues,thus showcasing the promising potential for future applications in orbital and maxillofacial surgery.
基金Supported by The Hunan Provincial Natural Science Foundation of China,No.2023JJ30773,No.2025JJ60480,and No.2025JJ60552The Scientific Research Program of The Hunan Provincial Health Commission,No.202204072544+4 种基金The Science and Technology Innovation Program of Hunan Province,No.2024RC3053The CBT ECR/MCR Scheme,No.324910-0028/07National Natural Science Foundation of China,No.32300652The Scientific Research Program of Hunan Provincial Health Commission,No.W20243023The Scientific Research Launch Project for New Employees of The Second Xiangya Hospital of Central South University.
文摘Augmented reality(AR)is a technology that superimposes digital information onto real-world objects via head-mounted display devices to improve surgical finesse through visually enhanced medical information.With the rapid development of digital technology,AR has been increasingly adopted in orthopedic surgeries across the globe,especially in total knee arthroplasty procedures which demand high precision.By overlaying digital information onto the surgeon's field of view,AR systems enhance precision,improve alignment accuracy,and reduce the risk of complications associated with malalignment.Some concerns have been raised despite accuracy,including the learning curve,long-term outcomes,and technical limitations.Furthermore,it is essential for health practitioners to gain trust in the utilisation of AR.
基金the National Natural Science Foundation of China(No.11502146)the 1 Batch of 2021 MOE of PRC Industry University Collaborative Education Program(No.202101042008)。
文摘The aim of this study was to assess the potential of surgical guides as a complementary tool to augmented reality(AR)in enhancing the safety and precision of pedicle screw placement in spinal surgery.Four trainers were divided into the AR navigation group using surgical guides and the free-hand group.Each group consisted of a novice and an experienced spine surgeon.A total of 80 pedicle screws were implanted.First,the AR group reconstructed the 3D model and planned the screw insertion route according to the computed tomography data of L2 lumbar vertebrae.Then,the Microsoft HoloLens™2 was used to identify the vertebral model,and the planned virtual path was superimposed on the real cone model.Next,the screw was placed according to the projected trajectory.Finally,Micron Tracker was used to measure the deviation of screws from the preoperatively planned trajectory,and pedicle screws were evaluated using the Gertzbein-Robbins scale.In the AR group,the linear deviations of the experienced doctor and the novice were(1.59±0.39)mm and(1.73±0.52)mm respectively,and the angle deviations were 2.72°±0.61°and 2.87°±0.63°respectively.In the free-hand group,the linear deviations of the experienced doctor and the novice were(2.88±0.58)mm and(5.25±0.62)mm respectively,and the angle deviations were 4.41°±1.18°and 7.15°±1.45°respectively.Both kinds of deviations between the two groups were significantly different(P<0.05).The screw accuracy rate was 95%in the AR navigation group and 77.5%in the free-hand group.The results of this study indicate that the integration of surgical guides and AR is an innovative technique that can substantially enhance the safety and precision of spinal surgery and assist inexperienced doctors in completing the surgery.
基金the University of Shanghai for Science and Technology’s Medical Engineering Interdisciplinary Project(No.10-22-308-520)the Ministry of Education’s First Batch of Industry-Education Cooperation Collaborative Education Projects(No.202101042008)+1 种基金the Fundamental Research Funds for the Central Universities(No.YG2019QNA34)the Shanghai Municipal Health Commission for Youth Clinical Research Project(No.20194Y0134)。
文摘The purpose of this study is to establish a multivariate nonlinear regression mathematical model to predict the displacement of tumor during brain tumor resection surgery.And the study will be integrated with augmented reality technology to achieve three-dimensional visualization,thereby enhancing the complete resection rate of tumor and the success rate of surgery.Based on the preoperative MRI data of the patients,a 3D virtual model is reconstructed and 3D printed.A brain biomimetic model is created using gel injection molding.By considering cerebrospinal fluid loss and tumor cyst fluid loss as independent variables,the highest point displacement in the vertical bone window direction is determined as the dependent variable after positioning the patient for surgery.An orthogonal experiment is conducted on the biomimetic model to establish a predictive model,and this model is incorporated into the augmented reality navigation system.To validate the predictive model,five participants wore HoloLens2 devices,overlaying the patient’s 3D virtual model onto the physical head model.Subsequently,the spatial coordinates of the tumor’s highest point after displacement were measured on both the physical and virtual models(actual coordinates and predicted coordinates,respectively).The difference between these coordinates represents the model’s prediction error.The results indicate that the measured and predicted errors for the displacement of the tumor’s highest point on the X and Y axes range from−0.6787 mm to 0.2957 mm and−0.4314 mm to 0.2253 mm,respectively.The relative errors for each experimental group are within 10%,demonstrating a good fit of the model.This method of establishing a regression model represents a preliminary attempt to predict brain tumor displacement in specific situations.It also provides a new approach for surgeons.By combining augmented reality visualization,it addresses the need for predicting tumor displacement and precisely locating brain anatomical structures in a simple and cost-effective manner.
文摘With the iteration and upgrading of medical technology and the continuous growth of public health demands,the quality of nursing services has become a core indicator for measuring the effectiveness of the medical system.The clinical practice ability of nursing staff is directly related to the safety of patient diagnosis and treatment and the rehabilitation process.However,the current clinical nursing talent training model is facing bottlenecks such as limited practical scenarios and fragmented case cognition.This study focuses on the teaching application of augmented reality(AR)technology in hospital Settings and systematically reviews the research progress on the improvement of clinical practice ability of trainee nurses based on the AR immersive teaching model.By constructing a clinical teaching scenario that integrates virtual and real,AR technology can dynamically simulate complex case handling processes and enhance nursing students’three-dimensional cognition of condition assessment,operation norms,and emergency plans.Hospitals,as the core base for practical teaching,can effectively shorten the connection cycle between theoretical teaching and clinical practice by integrating AR technology,improve the clinical practice level of trainee nurses,and provide an innovative model for optimizing the path of clinical nursing talent cultivation.
基金the Shanghai Municipal Education Commission-Gaofeng Clinical Medicine Grant(No.20172005)。
文摘Closed thoracic drainage can be performed using a steel-needle-guided chest tube to treat pleural effusion or pneumothorax in clinics.However,the puncture procedure during surgery is invisible,increasing the risk of surgical failure.Therefore,it is necessary to design a visualization system for closed thoracic drainage.Augmented reality(AR)technology can assist in visualizing the internal anatomical structure and determining the insertion point on the body surface.The structure of the currently used steel-needle-guided chest tube was modified by integrating it with an ultrafine diameter camera to provide real-time visualization of the puncture process.After simulation experiments,the overall registration error of the AR method was measured to be within(3.59±0.53)mm,indicating its potential for clinical application.The ultrafine diameter camera module and improved steel-needle-guided chest tube can timely reflect the position of the needle tip in the human body.A comparative experiment showed that video guidance could improve the safety of the puncture process compared to the traditional method.Finally,a qualitative evaluation of the usability of the system was conducted through a questionnaire.This system facilitates the visualization of closed thoracic drainage puncture procedure and pro-vides an implementation scheme to enhance the accuracy and safety of the operative step,which is conducive to reducing the learning curve and improving the proficiency of the doctors.
基金supported by grants from the Mission Plan Program of Beijing Municipal Administration of Hospitals(SML20152201)Beijing Municipal Administration of Hospitals Clinical Medicine Development of Special Funding(ZYLX201712)+1 种基金the National Natural Science Foundation of China(81427803)Beijing Tsinghua Changgung Hospital Fund(12015C1039)
文摘Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.Data Sources: The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the Pub Med database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.Results: In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery,which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.Conclusions: With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling,and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.
基金supported in part by the National Key R&D Program of China under Grant 2018YFE0205503in part by the National Natural Science Foundation of China (NSFC) under Grant 61671081+4 种基金in part by the Funds for International Cooperation and Exchange of NSFC under Grant 61720106007in part by the 111 Project under Grant B18008in part by the Beijing Natural Science Foundation under Grant 4172042in part by the Fundamental Research Funds for the Central Universities under Grant 2018XKJC01in part by the BUPT Excellent Ph.D. Students Foundation under Grant CX2019213
文摘The popularity of wearable devices and smartphones has fueled the development of Mobile Augmented Reality(MAR),which provides immersive experiences over the real world using techniques,such as computer vision and deep learning.However,the hardware-specific MAR is costly and heavy,and the App-based MAR requires an additional download and installation and it also lacks cross-platform ability.These limitations hamper the pervasive promotion of MAR.This paper argues that mobile Web AR(MWAR)holds the potential to become a practical and pervasive solution that can effectively scale to millions of end-users because MWAR can be developed as a lightweight,cross-platform,and low-cost solution for end-to-end delivery of MAR.The main challenges for making MWAR a reality lie in the low efficiency for dense computing in Web browsers,a large delay for real-time interactions over mobile networks,and the lack of standardization.The good news is that the newly emerging 5G and Beyond 5G(B5G)cellular networks can mitigate these issues to some extent via techniques such as network slicing,device-to-device communication,and mobile edge computing.In this paper,we first give an overview of the challenges and opportunities of MWAR in the 5G era.Then we describe our design and development of a generic service-oriented framework(called MWAR5)to provide a scalable,flexible,and easy to deploy MWAR solution.We evaluate the performance of our MWAR5 system in an actually deployed 5G trial network under the collaborative configurations,which shows encouraging results.Moreover,we also share the experiences and insights from our development and deployment,including some exciting future directions of MWAR over 5G and B5G networks.
基金supported by a Grant-in-Aid for Scientific Research (22659366) from the Japan Society for the Promotion of Science
文摘To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. Three-dimensional surface models of the jawbones, based on the computed tomography data, were used to create the integral videography images of a subject＇s maxillofacial area. The three-dimensional augmented reality system （integral videography display, computed tomography, a position tracker and a computer） was used to generate a three-dimensional overlay that was projected on the surgical site via a half-silvered mirror. Thereafter, a feasibility study was performed on a volunteer. The accuracy of this system was verified on a solid model while simulating bone resection. Positional registration was attained by identifying and tracking the patient/surgical instrument＇s position. Thus, integral videography images of jawbones, teeth and the surgical tool were superimposed in the correct position. Stereoscopic images viewed from various angles were accurately displayed. Change in the viewing angle did not negatively affect the surgeon＇s ability to simultaneously observe the three-dimensional images and the patient, without special glasses. The difference in three-dimensional position of each measuring point on the solid model and augmented reality navigation was almost negligible （〈1 mm）; this indicates that the system was highly accurate. This augmented reality system was highly accurate and effective for surgical navigation and for overlaying a three-dimensional computed tomography image on a patient＇s surgical area, enabling the surgeon to understand the positional relationship between the preoperative image and the actual surgical site, with the naked eye.
文摘Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass optical head-mounted display to train urology residents in how to place an inflatable penile prosthesis. We use the phrase Augmented Reality Assisted Surgery to describe this novel application of augmented reality in the setting of surgery. The application demonstrates the steps of the surgical procedure of inflatable penile prosthesis placement. It also contains software that allows for detection of interest points using a camera feed from the optical head-mounted display to enable faculty to interact with residents during placement of the penile prosthesis. Urology trainees and faculty who volunteered to take part in the study were given time to experience the technology in the operative or perioperative setting and asked to complete a feedback survey. From 30 total participants using a lO-point scale, educational usefulness was rated 8.6, ease of navigation was rated 7.6, likelihood to use was rated 7.4, and distraction in operating room was rated 4.9. When stratified between trainees and faculty, trainees found the technology more educationally useful, and less distracting. Overall, 81% of the participants want this technology in their residency program, and 93% see this technology in the operating room in the future. Further development of this technology is warranted before full release, and further studies are necessary to better characterize the effectiveness of Augmented Reality Assisted Surgery in urologic surgical training.
文摘The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.
基金supported by the National Natural Science Foundation of China (11904302)the Fundamental Research Funds for the Central Universities (Grant No.20720190005)+1 种基金the Major Science and Technology Project of Xiamen in China (3502Z20191015)Hong Kong University of Science and Technology-Foshan Joint Research Program (FSUST19-FYTRI11)
文摘Augmented reality(AR)and virtual reality(VR)are two novel display technologies that are under updates.The essential feature of AR/VR is the full-color display that requires high pixel densities.To generate three-color pixels,the fluorescent color conversion layer inevitably includes green and red pixels.To fabricate such sort of display kits,inkjet printing is a promising way to position the color conversion layers.In this review article,the progress of AR/VR technologies is first reviewed,and in succession,the state of the art of inkjet printing,as well as two key issues-the optimization of ink and the reduction of coffee-ring effects,are introduced.Finally,some potential problems associated with the color converting layer are highlighted.
基金supported by National Natural Science Foundation of China(No.61125101).
文摘Tracking registration is a key issue in augmented reality applications,particularly where there are no artificial identifier placed manually.In this paper,an efficient markerless tracking registration algorithm which combines the detector and the tracker is presented for the augmented reality system.We capture the target images in real scenes as template images,use the random ferns classifier for target detection and solve the problem of reinitialization after tracking registration failures due to changes in ambient lighting or occlusion of targets.Once the target has been successfully detected,the pyramid Lucas-Kanade(LK)optical flow tracker is used to track the detected target in real time to solve the problem of slow speed.The least median of squares(LMedS)method is used to adaptively calculate the homography matrix,and then the three-dimensional pose is estimated and the virtual object is rendered and registered.Experimental results demonstrate that the algorithm is more accurate,faster and more robust.
基金This work was supported by National Natural Science Foundation of China(No.61125101).
文摘In order to overcome the defects where the surface of the object lacks sufficient texture features and the algorithm cannot meet the real-time requirements of augmented reality,a markerless augmented reality tracking registration method based on multimodal template matching and point clouds is proposed.The method first adapts the linear parallel multi-modal LineMod template matching method with scale invariance to identify the texture-less target and obtain the reference image as the key frame that is most similar to the current perspective.Then,we can obtain the initial pose of the camera and solve the problem of re-initialization because of tracking registration interruption.A point cloud-based method is used to calculate the precise pose of the camera in real time.In order to solve the problem that the traditional iterative closest point(ICP)algorithm cannot meet the real-time requirements of the system,Kdtree(k-dimensional tree)is used under the graphics processing unit(GPU)to replace the part of finding the nearest points in the original ICP algorithm to improve the speed of tracking registration.At the same time,the random sample consensus(RANSAC)algorithm is used to remove the error point pairs to improve the accuracy of the algorithm.The results show that the proposed tracking registration method has good real-time performance and robustness.
文摘The massive technological advancements around the world have created significant challenging competition among companies where each of the companies tries to attract the customers using different techniques. One of the recent tech- niques is Augmented Reality (AR). The AR is a new technology which is capable of presenting possibilities that are difficult for other technologies to offer and meet. Nowadays, numerous augmented reality applications have been used in the industry of different kinds and disseminated all over the world. AR will really alter the way individuals view the world. The AR is yet in its initial phases of research and development at different colleges and high-tech institutes. Throughout the last years, AR apps became transportable and generally available on various devices. Besides, AR be- gins to occupy its place in our audio-visual media and to be used in various fields in our life in tangible and exciting ways such as news, sports and is used in many domains in our life such as electronic commerce, promotion, design, and business. In addition, AR is used to facilitate the learning whereas it enables students to access location-specific infor- mation provided through various sources. Such growth and spread of AR applications pushes organizations to compete one another, every one of them exerts its best to gain the customers. This paper provides a comprehensive study of AR including its history, architecture, applications, current challenges and future trends.
基金supported by Tianjin Sci-tech Planning Projects (14RCGFGX00846)Natural Science Foundation of Hebei Province (F2015202239)+1 种基金Tianjin Sci-tech Planning Projects (15ZCZDNC00130)Science and Technology Research Project of Hebei Province (Z2015044)
基金Project supported by Science Foundation of Shanghai Municipal Commission of Science and Technology (Grant No .025115008)
文摘Nonlinear errors always exist in data obtained from tracker in augmented reality （AR）, which badly influence the effect of AR. This paper proposes to rectify the errors using BP neural network. As BP neural network is prone to getting into local extrema and convergence is slow, genetic algorithm is employed to optimize the initial weights and threshold of neural network. This paper discusses how to set the crucial parameters in the algorithm. Experimental results show that the method ensures that the neural network achieves global convergence quickly and correctly. Tracking precision of AR system is improved after the tracker is rectified, and the third dimension of AR system is enhanced.
