Three-dimensional(3D)modeling is an important topic in computer graphics and computer vision.In recent years,the introduction of consumer-grade depth cameras has resulted in profound advances in 3D modeling.Starting w...Three-dimensional(3D)modeling is an important topic in computer graphics and computer vision.In recent years,the introduction of consumer-grade depth cameras has resulted in profound advances in 3D modeling.Starting with the basic data structure,this survey reviews the latest developments of 3D modeling based on depth cameras,including research works on camera tracking,3D object and scene reconstruction,and high-quality texture reconstruction.We also discuss the future work and possible solutions for 3D modeling based on the depth camera.展开更多
The aim of this study is to propose a novel system that has an ability to detect intra-fractional motion during radiotherapy treatment in real-time using three-dimensional surface taken by a depth camera, Microsoft Ki...The aim of this study is to propose a novel system that has an ability to detect intra-fractional motion during radiotherapy treatment in real-time using three-dimensional surface taken by a depth camera, Microsoft Kinect v1. Our approach introduces three new aspects for three-dimensional surface tracking in radiotherapy treatment. The first aspect is a new algorithm for noise reduction of depth values. Ueda’s algorithm was implemented and enabling a fast least square regression of depth values. The second aspect is an application for detection of patient’s motion at multiple points in thracoabdominal regions. The third aspect is an estimation of three-dimensional surface from multiple depth values. For evaluation of noise reduction by Ueda’s algorithm, two respiratory patterns are measured by the Kinect as well as a laser range meter. The resulting cross correlation coefficients between the laser range meter and the Kinect were 0.982 for abdominal respiration and 0.995 for breath holding. Moreover, the mean cross correlation coefficients between the signals of our system and the signals of Anzai with respect to participant’s respiratory motion were 0.90 for thoracic respiration and 0.93 for abdominal respiration, respectively. These results proved that the performance of the developed system was comparable to existing motion monitoring devices. Reconstruction of three-dimensional surface also enabled us to detect the irregular motion and breathing arrest by comparing the averaged depth with predefined threshold values.展开更多
With many advantages such as non-invasive,safe and quick effect,focused ultrasound lipolysis stands out among many fat-removing methods.However,during the whole process,the doctor needs to hold the ultrasound transduc...With many advantages such as non-invasive,safe and quick effect,focused ultrasound lipolysis stands out among many fat-removing methods.However,during the whole process,the doctor needs to hold the ultrasound transducer and press it on the patient’s skin with a large pressure for a long time;thus the probability of muscle and bone damage for doctors is greatly increased.To reduce the occurrence of doctors’occupational diseases,a depth camera-based ultrasonic lipolysis robot system is proposed to realize robot-assisted automatic ultrasonic lipolysis operation.The system is composed of RealSense depth camera,KUKA LBR Med seven-axis robotic arm,PC host,and ultrasonic lipolysis instrument.The whole operation includes two parts:preoperative planning and intraoperative operation.In preoperative planning,the treatment area is selected in the camera image by the doctor;then the system automatically plans uniformly distributed treatment points in the treatment area.At the same time,the skin normal vector is calculated to determine the end posture of the robot,so that the ultrasound transducer can be pressed down in the normal direction of skin.During the intraoperative operation,the robot is controlled to arrive at the treatment point in turn.Meanwhile,the patient’s movement can be detected by the depth camera,and the path of robot is adjusted in real time so that the robot can track the movement of patient,thereby ensuring the accuracy of the ultrasonic lipolysis operation.Finally,the human body model experiment is conducted.The results show that the maximum error of the robot operation is within 5mm,average error is 3.1mm,and the treatment points of the robot operation are more uniform than those of manual operation.Therefore,the system can replace the doctor and achieve autonomous ultrasonic lipolysis to reduce the doctor’s labor intensity.展开更多
This paper proposes a depth measurement error model for consumer depth cameras such as the Microsoft Kinect, and a corresponding calibration method. These devices were originally designed as video game interfaces, and...This paper proposes a depth measurement error model for consumer depth cameras such as the Microsoft Kinect, and a corresponding calibration method. These devices were originally designed as video game interfaces, and their output depth maps usually lack sufficient accuracy for 3 D measurement.Models have been proposed to reduce these depth errors, but they only consider camera-related causes.Since the depth sensors are based on projectorcamera systems, we should also consider projectorrelated causes. Also, previous models require disparity observations, which are usually not output by such sensors, so cannot be employed in practice. We give an alternative error model for projector-camera based consumer depth cameras, based on their depth measurement algorithm, and intrinsic parameters of the camera and the projector; it does not need disparity values. We also give a corresponding new parameter estimation method which simply needs observation of a planar board. Our calibrated error model allows use of a consumer depth sensor as a 3 D measuring device.Experimental results show the validity and effectiveness of the error model and calibration procedure.展开更多
Fast assessment of the initial carbon to nitrogen ratio(C/N)of organic fraction of municipal solid waste(OFMSW)is an important prerequisite for automatic composting control to improve efficiency and stability of the b...Fast assessment of the initial carbon to nitrogen ratio(C/N)of organic fraction of municipal solid waste(OFMSW)is an important prerequisite for automatic composting control to improve efficiency and stability of the bioconversion process.In this study,a novel approach was proposed to estimate the C/N of OFMSW,where an instance segmentation model was applied to predict the masks for the waste images.Then,by combining the instance segmentation model with the depth-camera-based volume calculation algorithm,the volumes occupied by each type of waste were obtained,therefore the C/N could be estimated based on the properties of each type of waste.First,an instance segmentation dataset including three common classes of OFMSW was built to train mask region-based convolutional neural networks(Mask R-CNN)model.Second,a volume measurement algorithm was proposed,where the measurement result of the object was derived by accumulating the volumes of small rectangular cuboids whose bottom area was calculated with the projection property.Then the calculated volume was corrected with linear regression models.The results showed that the trained instance segmentation model performed well with average precision scores AP_(50)=82.9,AP_(75)=72.5,and mask intersection over unit(Mask IoU)=45.1.A high correlation was found between the estimated C/N and the ground truth with a coefficient of determination R2=0.97 and root mean square error RMSE=0.10.The relative average error was 0.42%and the maximum error was only 1.71%,which indicated this approach has potential for practical applications.展开更多
Purpose:To evaluate the relationship between the position of the focal adjustment knob of a fundus camera and refractive error and biometric data as measured in the same eye.Methods:Normal eyes of patients presenting ...Purpose:To evaluate the relationship between the position of the focal adjustment knob of a fundus camera and refractive error and biometric data as measured in the same eye.Methods:Normal eyes of patients presenting to clinics at the Beijing Tongren Hospital were examined with a non-mydriatic fundus camera.The position on the focal scale of a knob adjusting the distance between the camera lens and film plane,used to adjust focus the image of the patients fundus relative to the refractive power of the eye,was recorded in degrees.Ocular biometry and refractometry were performed on the same eyes.Results:The study included 136 subjects with a mean age of 36.5 ±19.6 years and a mean refractive error of-1.31 ±2.77 diopters.In univariate analysis,the position of the adjustment knob was significantly associated with refractive error.(P < 0.001;correlation coefficient r=-0.77),axial length.(P<0.001;r=0.65) and anterior chamber depth (P<0.001;r=0.48).After adjustment for age,anterior chamber depth decreased by 0.01 mm(95% confidence interval:0.003,0.017) for change per degree in the position of the adjustment knob.Conclusion:A fundus camera can be used to estimate anterior chamber depth,axial length and refractive error.In a screening setting,a fundus camera operated by a technician may be helpful to detect a shallow anterior chamber and evaluate a potential risk factor for primary angle closure.展开更多
High performance hardware architecture for depth measurement by using binocular-camera is proposed.In the system,at first,video streams of the target are captured by left and right charge-coupled device(CCD)cameras to...High performance hardware architecture for depth measurement by using binocular-camera is proposed.In the system,at first,video streams of the target are captured by left and right charge-coupled device(CCD)cameras to obtain an image including the target.Then,two different images with two different view points are obtained,and they are used in calculating the position deviation of the image's pixels based on triangular measurement.Finally,the three-dimensional coordinate of the object is reconstructed.All the video data is processed by using field-programmable gate array(FPGA)in real-time.Hardware implementation speeds up the performance and reduces the power,thus,this hardware architecture can be applied in the portable environment.展开更多
间接飞行时间(indirect time of flight,iTOF)相机在三维环境感知领域有着广泛的应用前景。根据iTOF相机成像原理,当曝光时间过大导致相机工作在非线性区时,解算的深度信息也会引入额外的偏差,从而影响测量准确度。为了进一步提高飞行...间接飞行时间(indirect time of flight,iTOF)相机在三维环境感知领域有着广泛的应用前景。根据iTOF相机成像原理,当曝光时间过大导致相机工作在非线性区时,解算的深度信息也会引入额外的偏差,从而影响测量准确度。为了进一步提高飞行时间相机的应用精度,根据飞行时间相机的光学成像机理,提出一种针对飞行时间相机的性能参数测量方法,通过实验和计算间接得到飞行时间深度相机的系统增益等性能参数,将其代入飞行时间相机的光学成像模型,即可得到距离与相机输出灰度值的对应曲线。建模与实验结果的相对误差可以达到20%以内,平均相对误差为0.16%。这些性能参数和仿真模型用于指导iTOF相机在不同场景使用时的积分时间选择,可以有效解决因使用不当引入非线性误差而降低距离测量精度的问题。展开更多
传统松科球果采摘面临效率低、风险高和成本不可控等挑战,针对自动化松科球果采摘对果实的实时识别与定位问题,提出改进的YOLOv5s-7.0(You Only Look Once)目标检测模型,基于此模型,构建基于双目深度相机的松科球果检测与定位网络。为...传统松科球果采摘面临效率低、风险高和成本不可控等挑战,针对自动化松科球果采摘对果实的实时识别与定位问题,提出改进的YOLOv5s-7.0(You Only Look Once)目标检测模型,基于此模型,构建基于双目深度相机的松科球果检测与定位网络。为提高目标检测精度及效率,对YOLOv5s模型进行改进,将部分卷积PConv嵌入到模型的颈部网络neck多分枝堆叠结构中,面对松科球果的复杂场景增强对稀疏特征的处理能力,提升鲁棒性,减轻特征信息的冗余。在骨干网络backbone的深层及backbone与neck的连接处嵌入简单注意力机制SimAM,在不引入过多参数的基础上优化模型复杂背景下特征提取能力和信息传递的有效性。为满足高效率检测定位,基于双目深度相机测距原理和改进的YOLOv5s模型搭建目标检测及实时定位代码,通过深度匹配,构建松科球果检测与定位系统。根据构建的大兴安岭樟子松球果与小兴安岭红松球果数据集,改进后YOLOv5s模型目标检测精确率达96.8%,召回率和平均精度分别达94%、96.3%,松科球果检测与定位系统在x轴、y轴、z轴的平均绝对误差分别为0.644、0.620、0.740 cm,顺、侧、逆光照下定位试验成功率93.3%,暗光下定位成功率83.3%,视场角等其他性能符合松科球果采摘需求。研究提出的松科球果检测与定位系统为机械化采摘的实时目标检测与定位问题提供可靠的解决方案。展开更多
基金National Natural Science Foundation of China(61732016).
文摘Three-dimensional(3D)modeling is an important topic in computer graphics and computer vision.In recent years,the introduction of consumer-grade depth cameras has resulted in profound advances in 3D modeling.Starting with the basic data structure,this survey reviews the latest developments of 3D modeling based on depth cameras,including research works on camera tracking,3D object and scene reconstruction,and high-quality texture reconstruction.We also discuss the future work and possible solutions for 3D modeling based on the depth camera.
文摘The aim of this study is to propose a novel system that has an ability to detect intra-fractional motion during radiotherapy treatment in real-time using three-dimensional surface taken by a depth camera, Microsoft Kinect v1. Our approach introduces three new aspects for three-dimensional surface tracking in radiotherapy treatment. The first aspect is a new algorithm for noise reduction of depth values. Ueda’s algorithm was implemented and enabling a fast least square regression of depth values. The second aspect is an application for detection of patient’s motion at multiple points in thracoabdominal regions. The third aspect is an estimation of three-dimensional surface from multiple depth values. For evaluation of noise reduction by Ueda’s algorithm, two respiratory patterns are measured by the Kinect as well as a laser range meter. The resulting cross correlation coefficients between the laser range meter and the Kinect were 0.982 for abdominal respiration and 0.995 for breath holding. Moreover, the mean cross correlation coefficients between the signals of our system and the signals of Anzai with respect to participant’s respiratory motion were 0.90 for thoracic respiration and 0.93 for abdominal respiration, respectively. These results proved that the performance of the developed system was comparable to existing motion monitoring devices. Reconstruction of three-dimensional surface also enabled us to detect the irregular motion and breathing arrest by comparing the averaged depth with predefined threshold values.
基金the National Natural Science Foundation of China(Nos.61973211,51911540479 and M-0221)the Research Project of Institute of Medical Robotics of Shanghai Jiao Tong Universitythe Project of Science and Technology Commission of Shanghai Municipality(No.20DZ2220400)。
文摘With many advantages such as non-invasive,safe and quick effect,focused ultrasound lipolysis stands out among many fat-removing methods.However,during the whole process,the doctor needs to hold the ultrasound transducer and press it on the patient’s skin with a large pressure for a long time;thus the probability of muscle and bone damage for doctors is greatly increased.To reduce the occurrence of doctors’occupational diseases,a depth camera-based ultrasonic lipolysis robot system is proposed to realize robot-assisted automatic ultrasonic lipolysis operation.The system is composed of RealSense depth camera,KUKA LBR Med seven-axis robotic arm,PC host,and ultrasonic lipolysis instrument.The whole operation includes two parts:preoperative planning and intraoperative operation.In preoperative planning,the treatment area is selected in the camera image by the doctor;then the system automatically plans uniformly distributed treatment points in the treatment area.At the same time,the skin normal vector is calculated to determine the end posture of the robot,so that the ultrasound transducer can be pressed down in the normal direction of skin.During the intraoperative operation,the robot is controlled to arrive at the treatment point in turn.Meanwhile,the patient’s movement can be detected by the depth camera,and the path of robot is adjusted in real time so that the robot can track the movement of patient,thereby ensuring the accuracy of the ultrasonic lipolysis operation.Finally,the human body model experiment is conducted.The results show that the maximum error of the robot operation is within 5mm,average error is 3.1mm,and the treatment points of the robot operation are more uniform than those of manual operation.Therefore,the system can replace the doctor and achieve autonomous ultrasonic lipolysis to reduce the doctor’s labor intensity.
基金supported by the JST CREST“Behavior Understanding based on Intention-Gait Model”project
文摘This paper proposes a depth measurement error model for consumer depth cameras such as the Microsoft Kinect, and a corresponding calibration method. These devices were originally designed as video game interfaces, and their output depth maps usually lack sufficient accuracy for 3 D measurement.Models have been proposed to reduce these depth errors, but they only consider camera-related causes.Since the depth sensors are based on projectorcamera systems, we should also consider projectorrelated causes. Also, previous models require disparity observations, which are usually not output by such sensors, so cannot be employed in practice. We give an alternative error model for projector-camera based consumer depth cameras, based on their depth measurement algorithm, and intrinsic parameters of the camera and the projector; it does not need disparity values. We also give a corresponding new parameter estimation method which simply needs observation of a planar board. Our calibrated error model allows use of a consumer depth sensor as a 3 D measuring device.Experimental results show the validity and effectiveness of the error model and calibration procedure.
基金funded by the National Key Research and Development Program of China(Grant No.2018YFD0200800)Key Research and Development Program of Hunan Province(Grant No.2018GK2013)+1 种基金Hunan Modern Agricultural Industry Technology Program(Grant No.201926)Innovation and Entrepreneurship Training Program of Hunan Agricultural University(Grant No.2019062x).
文摘Fast assessment of the initial carbon to nitrogen ratio(C/N)of organic fraction of municipal solid waste(OFMSW)is an important prerequisite for automatic composting control to improve efficiency and stability of the bioconversion process.In this study,a novel approach was proposed to estimate the C/N of OFMSW,where an instance segmentation model was applied to predict the masks for the waste images.Then,by combining the instance segmentation model with the depth-camera-based volume calculation algorithm,the volumes occupied by each type of waste were obtained,therefore the C/N could be estimated based on the properties of each type of waste.First,an instance segmentation dataset including three common classes of OFMSW was built to train mask region-based convolutional neural networks(Mask R-CNN)model.Second,a volume measurement algorithm was proposed,where the measurement result of the object was derived by accumulating the volumes of small rectangular cuboids whose bottom area was calculated with the projection property.Then the calculated volume was corrected with linear regression models.The results showed that the trained instance segmentation model performed well with average precision scores AP_(50)=82.9,AP_(75)=72.5,and mask intersection over unit(Mask IoU)=45.1.A high correlation was found between the estimated C/N and the ground truth with a coefficient of determination R2=0.97 and root mean square error RMSE=0.10.The relative average error was 0.42%and the maximum error was only 1.71%,which indicated this approach has potential for practical applications.
文摘Purpose:To evaluate the relationship between the position of the focal adjustment knob of a fundus camera and refractive error and biometric data as measured in the same eye.Methods:Normal eyes of patients presenting to clinics at the Beijing Tongren Hospital were examined with a non-mydriatic fundus camera.The position on the focal scale of a knob adjusting the distance between the camera lens and film plane,used to adjust focus the image of the patients fundus relative to the refractive power of the eye,was recorded in degrees.Ocular biometry and refractometry were performed on the same eyes.Results:The study included 136 subjects with a mean age of 36.5 ±19.6 years and a mean refractive error of-1.31 ±2.77 diopters.In univariate analysis,the position of the adjustment knob was significantly associated with refractive error.(P < 0.001;correlation coefficient r=-0.77),axial length.(P<0.001;r=0.65) and anterior chamber depth (P<0.001;r=0.48).After adjustment for age,anterior chamber depth decreased by 0.01 mm(95% confidence interval:0.003,0.017) for change per degree in the position of the adjustment knob.Conclusion:A fundus camera can be used to estimate anterior chamber depth,axial length and refractive error.In a screening setting,a fundus camera operated by a technician may be helpful to detect a shallow anterior chamber and evaluate a potential risk factor for primary angle closure.
文摘High performance hardware architecture for depth measurement by using binocular-camera is proposed.In the system,at first,video streams of the target are captured by left and right charge-coupled device(CCD)cameras to obtain an image including the target.Then,two different images with two different view points are obtained,and they are used in calculating the position deviation of the image's pixels based on triangular measurement.Finally,the three-dimensional coordinate of the object is reconstructed.All the video data is processed by using field-programmable gate array(FPGA)in real-time.Hardware implementation speeds up the performance and reduces the power,thus,this hardware architecture can be applied in the portable environment.
文摘间接飞行时间(indirect time of flight,iTOF)相机在三维环境感知领域有着广泛的应用前景。根据iTOF相机成像原理,当曝光时间过大导致相机工作在非线性区时,解算的深度信息也会引入额外的偏差,从而影响测量准确度。为了进一步提高飞行时间相机的应用精度,根据飞行时间相机的光学成像机理,提出一种针对飞行时间相机的性能参数测量方法,通过实验和计算间接得到飞行时间深度相机的系统增益等性能参数,将其代入飞行时间相机的光学成像模型,即可得到距离与相机输出灰度值的对应曲线。建模与实验结果的相对误差可以达到20%以内,平均相对误差为0.16%。这些性能参数和仿真模型用于指导iTOF相机在不同场景使用时的积分时间选择,可以有效解决因使用不当引入非线性误差而降低距离测量精度的问题。
文摘传统松科球果采摘面临效率低、风险高和成本不可控等挑战,针对自动化松科球果采摘对果实的实时识别与定位问题,提出改进的YOLOv5s-7.0(You Only Look Once)目标检测模型,基于此模型,构建基于双目深度相机的松科球果检测与定位网络。为提高目标检测精度及效率,对YOLOv5s模型进行改进,将部分卷积PConv嵌入到模型的颈部网络neck多分枝堆叠结构中,面对松科球果的复杂场景增强对稀疏特征的处理能力,提升鲁棒性,减轻特征信息的冗余。在骨干网络backbone的深层及backbone与neck的连接处嵌入简单注意力机制SimAM,在不引入过多参数的基础上优化模型复杂背景下特征提取能力和信息传递的有效性。为满足高效率检测定位,基于双目深度相机测距原理和改进的YOLOv5s模型搭建目标检测及实时定位代码,通过深度匹配,构建松科球果检测与定位系统。根据构建的大兴安岭樟子松球果与小兴安岭红松球果数据集,改进后YOLOv5s模型目标检测精确率达96.8%,召回率和平均精度分别达94%、96.3%,松科球果检测与定位系统在x轴、y轴、z轴的平均绝对误差分别为0.644、0.620、0.740 cm,顺、侧、逆光照下定位试验成功率93.3%,暗光下定位成功率83.3%,视场角等其他性能符合松科球果采摘需求。研究提出的松科球果检测与定位系统为机械化采摘的实时目标检测与定位问题提供可靠的解决方案。
