A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning...A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning level, a motion control level and a servo control level. The last three levels form a local real-time open robot controller, which realizes motion planning and motion control of a robot. A camera calibration method based on the relative movement of the end-effector connected to a robot is proposed and a method for tracking weld seam based on the structured light stereovision is provided. Combining the parameters of the cameras and laser plane, three groups of position values in Cartesian space are obtained for each feature point in a stripe projected on the weld seam. The accurate three-dimensional position of the edge points in the weld seam can be calculated from the obtained parameters with an information fusion algorithm. By calculating the weld seam parameter from position and image data, the movement parameters of the robot used for tracking can be determined. A swing welding experiment of type V groove weld is successfully conducted, the results of which show that the system has high resolution seam tracking in real-time, and works stably and efficiently.展开更多
This paper presents the principle and mathematic model for the 3D depth map method based on space encoding images performed by modulating scanning structuredlight according to time sequences,and the synchro control ...This paper presents the principle and mathematic model for the 3D depth map method based on space encoding images performed by modulating scanning structuredlight according to time sequences,and the synchro control among the camera,laser diode modulation and scanning polyhedron.展开更多
Based on the characteristics of line structured light sensor, a speedy method for the calibration was established. With the coplanar reference target, the spacial pose between camera and optical plane can be calibrate...Based on the characteristics of line structured light sensor, a speedy method for the calibration was established. With the coplanar reference target, the spacial pose between camera and optical plane can be calibrated by using of the camera’s projective center and the light’s information in the camera’s image surface. Without striction to the movement of the coplanar reference target and assistant adjustment equipment, this calibration method can be implemented. This method has been used and decreased the cost of calibration equipment, simplified the calibration procedure, improved calibration efficiency. Using experiment, the sensor can attain relative accuracy about 0.5%, which indicates the rationality and effectivity of this method.展开更多
A novel hybrid visual servoing control method based on structured light vision is pro-posed for robotic arc welding with a general six degrees of freedom robot. It consists of a positioncontrol inner-loop in Cartesian...A novel hybrid visual servoing control method based on structured light vision is pro-posed for robotic arc welding with a general six degrees of freedom robot. It consists of a positioncontrol inner-loop in Cartesian space and two outer-loops. One is position-based visual control inCartesian space for moving in the direction of weld seam, i.e., weld seam tracking, another is image-based visual control in image space for adjustment to eliminate the errors in the process of tracking.A new Jacobian matrix from image space of the feature point on structured light stripe to Cartesianspace is provided for dierential movement of the end-e?ector. The control system model is simplifiedand its stability is discussed. An experiment of arc welding protected by gas CO2 for verifying iswell conducted.展开更多
A calibration procedure was developed for three-dimensional(3D) binocular structured light measurement systems. In virtue of a specially designed pattern, matching points in stereo images are extracted. And then suffi...A calibration procedure was developed for three-dimensional(3D) binocular structured light measurement systems. In virtue of a specially designed pattern, matching points in stereo images are extracted. And then sufficient 3D space points are obtained through pairs of images with the intrinsic and extrinsic parameters of each camera estimated prior and consequently some lights are calibrated by means of multi point fitting. Finally, a mathematical model is applied to interpolate and approximate all dynamic scanning lights based on geometry. The process of calibration method is successfully used in the binocular 3D measurement system based on structured lights and the 3D reconstruction results are satisfying.展开更多
Three dimensional digitization of human head is desired in many applications. In this paper, an information fusion based scheme is presented to obtain 3-D information of human head. Structured light technology is empl...Three dimensional digitization of human head is desired in many applications. In this paper, an information fusion based scheme is presented to obtain 3-D information of human head. Structured light technology is employed to measure depth. For the special reflection areas,in which the structured light stripe can not be detected directly, the shape of the structured light stripe can be calculated from the corresponding contour. By fusing the information of structured light and the contours, the problem of reflectance influence is solved, and the whole shape of head,including hair area, can be obtained. Some good results are obtained.展开更多
In this study, a three-dimensional (3D) in-situ laser machining system integrating laser measurement and machining was built using a 3D galvanometer scanner equipped with a side-axis industrial camera. A line structur...In this study, a three-dimensional (3D) in-situ laser machining system integrating laser measurement and machining was built using a 3D galvanometer scanner equipped with a side-axis industrial camera. A line structured light measurement model based on a galvanometer scanner was proposed to obtain the 3D information of the workpiece. A height calibration method was proposed to further ensure measurement accuracy, so as to achieve accurate laser focusing. In-situ machining software was developed to realize time-saving and labor-saving 3D laser processing. The feasibility and practicability of this in-situ laser machining system were verified using specific cases. In comparison with the conventional line structured light measurement method, the proposed methods do not require light plane calibration, and do not need additional motion axes for 3D reconstruction;thus they provide technical and cost advantages. The insitu laser machining system realizes a simple operation process by integrating measurement and machining,which greatly reduces labor and time costs.展开更多
Nonlinear response is an important factor affecting the accuracy of three-dimensional image measurement based on the fringe structured light method.A phase compensation algorithm combined with a Hilbert transform is p...Nonlinear response is an important factor affecting the accuracy of three-dimensional image measurement based on the fringe structured light method.A phase compensation algorithm combined with a Hilbert transform is proposed to reduce the phase error caused by the nonlinear response of a digital projector in the three-dimensional measurement system of fringe structured light.According to the analysis of the influence of Gamma distortion on the phase calculation,the algorithm establishes the relationship model between phase error and harmonic coefficient,introduces phase shift to the signal,and keeps the signal amplitude constant while filtering out the DC component.The phase error is converted to the transform domain,and compared with the numeric value in the space domain.The algorithm is combined with a spiral phase function to optimize the Hilbert transform,so as to eliminate external noise,enhance the image quality,and get an accurate phase value.Experimental results show that the proposed method can effectively improve the accuracy and speed of phase measurement.By performing phase error compensation for free-form surface objects,the phase error is reduced by about 26%,and about 27%of the image reconstruction time is saved,which further demonstrates the feasibility and effectiveness of the method.展开更多
Three-dimensional(3D)imaging with structured light is crucial in diverse scenarios,ranging from intelligent manufacturing and medicine to entertainment.However,current structured light methods rely on projector-camera...Three-dimensional(3D)imaging with structured light is crucial in diverse scenarios,ranging from intelligent manufacturing and medicine to entertainment.However,current structured light methods rely on projector-camera synchronization,limiting the use of affordable imaging devices and their consumer applications.In this work,we introduce an asynchronous structured light imaging approach based on generative deep neural networks to relax the synchronization constraint,accomplishing the challenges of fringe pattern aliasing,without relying on any a priori constraint of the projection system.To overcome this need,we propose a generative deep neural network with U-Net-like encoder-decoder architecture to learn the underlying fringe features directly by exploring the intrinsic prior principles in the fringe pattern aliasing.We train within an adversarial learning framework and supervise the network training via a statisticsinformed loss function.We demonstrate that by evaluating the performance on fields of intensity,phase,and 3D reconstruction.It is shown that the trained network can separate aliased fringe patterns for producing comparable results with the synchronous one:the absolute error is no greater than 8μm,and the standard deviation does not exceed 3μm.Evaluation results on multiple objects and pattern types show it could be generalized for any asynchronous structured light scene.展开更多
The measurement of position and attitude parameters for the isolated target from a highspeed aircraft is a great challenge in the field of wind tunnel simulation technology.In this paper,firstly, an image acquisition ...The measurement of position and attitude parameters for the isolated target from a highspeed aircraft is a great challenge in the field of wind tunnel simulation technology.In this paper,firstly, an image acquisition method for small high-speed targets with multi-dimensional movement in wind tunnel environment is proposed based on laser-aided vision technology.Combining with the trajectory simulation of the isolated model, the reasonably distributed laser stripes and selfluminous markers are utilized to capture clear images of the object.Then, after image processing,feature extraction, stereo correspondence and reconstruction, three-dimensional information of laser stripes and self-luminous markers are calculated.Besides, a pose solution method based on projected laser stripes and self-luminous markers is proposed.Finally, simulation experiments on measuring the position and attitude of high-speed rolling targets are conducted, as well as accuracy verification experiments.Experimental results indicate that the proposed method is feasible and efficient for measuring the pose parameters of rolling targets in wind tunnels.展开更多
This paper theoretically analyzes and researches the coordinate frames of a 3D vision scanning system, establishes the mathematic model of a system scanning process, derives the relationship between the general non-or...This paper theoretically analyzes and researches the coordinate frames of a 3D vision scanning system, establishes the mathematic model of a system scanning process, derives the relationship between the general non-orthonormal sensor coordinate system and the machine coordinate system and the coordinate transformation matrix of the extrinsic calibration for the system.展开更多
Existingfirefighting robots are focused on simple storage orfire sup-pression outside buildings rather than detection or recognition.Utilizing a large number of robots using expensive equipment is challenging.This study ...Existingfirefighting robots are focused on simple storage orfire sup-pression outside buildings rather than detection or recognition.Utilizing a large number of robots using expensive equipment is challenging.This study aims to increase the efficiency of search and rescue operations and the safety offirefigh-ters by detecting and identifying the disaster site by recognizing collapsed areas,obstacles,and rescuers on-site.A fusion algorithm combining a camera and three-dimension light detection and ranging(3D LiDAR)is proposed to detect and loca-lize the interiors of disaster sites.The algorithm detects obstacles by analyzingfloor segmentation and edge patterns using a mask regional convolutional neural network(mask R-CNN)features model based on the visual data collected from a parallelly connected camera and 3D LiDAR.People as objects are detected using you only look once version 4(YOLOv4)in the image data to localize persons requiring rescue.The point cloud data based on 3D LiDAR cluster the objects using the density-based spatial clustering of applications with noise(DBSCAN)clustering algorithm and estimate the distance to the actual object using the center point of the clustering result.The proposed artificial intelligence(AI)algorithm was verified based on individual sensors using a sensor-mounted robot in an actual building to detectfloor surfaces,atypical obstacles,and persons requiring rescue.Accordingly,the fused AI algorithm was comparatively verified.展开更多
基金This work was supported by the National High Technology Research and Development Program of China under Grant 2002AA422160 by the National Key Fundamental Research and the Devel-opment Project of China (973) under Grant 2002CB312200.
文摘A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning level, a motion control level and a servo control level. The last three levels form a local real-time open robot controller, which realizes motion planning and motion control of a robot. A camera calibration method based on the relative movement of the end-effector connected to a robot is proposed and a method for tracking weld seam based on the structured light stereovision is provided. Combining the parameters of the cameras and laser plane, three groups of position values in Cartesian space are obtained for each feature point in a stripe projected on the weld seam. The accurate three-dimensional position of the edge points in the weld seam can be calculated from the obtained parameters with an information fusion algorithm. By calculating the weld seam parameter from position and image data, the movement parameters of the robot used for tracking can be determined. A swing welding experiment of type V groove weld is successfully conducted, the results of which show that the system has high resolution seam tracking in real-time, and works stably and efficiently.
文摘This paper presents the principle and mathematic model for the 3D depth map method based on space encoding images performed by modulating scanning structuredlight according to time sequences,and the synchro control among the camera,laser diode modulation and scanning polyhedron.
文摘Based on the characteristics of line structured light sensor, a speedy method for the calibration was established. With the coplanar reference target, the spacial pose between camera and optical plane can be calibrated by using of the camera’s projective center and the light’s information in the camera’s image surface. Without striction to the movement of the coplanar reference target and assistant adjustment equipment, this calibration method can be implemented. This method has been used and decreased the cost of calibration equipment, simplified the calibration procedure, improved calibration efficiency. Using experiment, the sensor can attain relative accuracy about 0.5%, which indicates the rationality and effectivity of this method.
文摘A novel hybrid visual servoing control method based on structured light vision is pro-posed for robotic arc welding with a general six degrees of freedom robot. It consists of a positioncontrol inner-loop in Cartesian space and two outer-loops. One is position-based visual control inCartesian space for moving in the direction of weld seam, i.e., weld seam tracking, another is image-based visual control in image space for adjustment to eliminate the errors in the process of tracking.A new Jacobian matrix from image space of the feature point on structured light stripe to Cartesianspace is provided for dierential movement of the end-e?ector. The control system model is simplifiedand its stability is discussed. An experiment of arc welding protected by gas CO2 for verifying iswell conducted.
基金The National Natural Science Foundation of China (No30470488)
文摘A calibration procedure was developed for three-dimensional(3D) binocular structured light measurement systems. In virtue of a specially designed pattern, matching points in stereo images are extracted. And then sufficient 3D space points are obtained through pairs of images with the intrinsic and extrinsic parameters of each camera estimated prior and consequently some lights are calibrated by means of multi point fitting. Finally, a mathematical model is applied to interpolate and approximate all dynamic scanning lights based on geometry. The process of calibration method is successfully used in the binocular 3D measurement system based on structured lights and the 3D reconstruction results are satisfying.
基金Supported by the National Natural Science Foundation of China(69775022) and 863 Programme of China(863-306-ZT04-06-3)
文摘Three dimensional digitization of human head is desired in many applications. In this paper, an information fusion based scheme is presented to obtain 3-D information of human head. Structured light technology is employed to measure depth. For the special reflection areas,in which the structured light stripe can not be detected directly, the shape of the structured light stripe can be calculated from the corresponding contour. By fusing the information of structured light and the contours, the problem of reflectance influence is solved, and the whole shape of head,including hair area, can be obtained. Some good results are obtained.
文摘In this study, a three-dimensional (3D) in-situ laser machining system integrating laser measurement and machining was built using a 3D galvanometer scanner equipped with a side-axis industrial camera. A line structured light measurement model based on a galvanometer scanner was proposed to obtain the 3D information of the workpiece. A height calibration method was proposed to further ensure measurement accuracy, so as to achieve accurate laser focusing. In-situ machining software was developed to realize time-saving and labor-saving 3D laser processing. The feasibility and practicability of this in-situ laser machining system were verified using specific cases. In comparison with the conventional line structured light measurement method, the proposed methods do not require light plane calibration, and do not need additional motion axes for 3D reconstruction;thus they provide technical and cost advantages. The insitu laser machining system realizes a simple operation process by integrating measurement and machining,which greatly reduces labor and time costs.
基金This work is funded by the Scientific and Technological Projects of Henan Province under Grant 152102210115.
文摘Nonlinear response is an important factor affecting the accuracy of three-dimensional image measurement based on the fringe structured light method.A phase compensation algorithm combined with a Hilbert transform is proposed to reduce the phase error caused by the nonlinear response of a digital projector in the three-dimensional measurement system of fringe structured light.According to the analysis of the influence of Gamma distortion on the phase calculation,the algorithm establishes the relationship model between phase error and harmonic coefficient,introduces phase shift to the signal,and keeps the signal amplitude constant while filtering out the DC component.The phase error is converted to the transform domain,and compared with the numeric value in the space domain.The algorithm is combined with a spiral phase function to optimize the Hilbert transform,so as to eliminate external noise,enhance the image quality,and get an accurate phase value.Experimental results show that the proposed method can effectively improve the accuracy and speed of phase measurement.By performing phase error compensation for free-form surface objects,the phase error is reduced by about 26%,and about 27%of the image reconstruction time is saved,which further demonstrates the feasibility and effectiveness of the method.
基金funding from the National Natural Science Foundation of China(Grant Nos.62375078 and 12002197)the Youth Talent Launching Program of Shanghai University+2 种基金the General Science Foundation of Henan Province(Grant No.222300420427)the Key Research Project Plan for Higher Education Institutions in Henan Province(Grant No.24ZX011)the National Key Laboratory of Ship Structural Safety
文摘Three-dimensional(3D)imaging with structured light is crucial in diverse scenarios,ranging from intelligent manufacturing and medicine to entertainment.However,current structured light methods rely on projector-camera synchronization,limiting the use of affordable imaging devices and their consumer applications.In this work,we introduce an asynchronous structured light imaging approach based on generative deep neural networks to relax the synchronization constraint,accomplishing the challenges of fringe pattern aliasing,without relying on any a priori constraint of the projection system.To overcome this need,we propose a generative deep neural network with U-Net-like encoder-decoder architecture to learn the underlying fringe features directly by exploring the intrinsic prior principles in the fringe pattern aliasing.We train within an adversarial learning framework and supervise the network training via a statisticsinformed loss function.We demonstrate that by evaluating the performance on fields of intensity,phase,and 3D reconstruction.It is shown that the trained network can separate aliased fringe patterns for producing comparable results with the synchronous one:the absolute error is no greater than 8μm,and the standard deviation does not exceed 3μm.Evaluation results on multiple objects and pattern types show it could be generalized for any asynchronous structured light scene.
基金supported by the National Natural Science Foundation of China (Nos.51375075, 51227004)the Scientific Research Fund of Liaoning Provincial Education Department of China (No.L2013035)the Science Fund for Creative Research Groups of China (No.51321004)
文摘The measurement of position and attitude parameters for the isolated target from a highspeed aircraft is a great challenge in the field of wind tunnel simulation technology.In this paper,firstly, an image acquisition method for small high-speed targets with multi-dimensional movement in wind tunnel environment is proposed based on laser-aided vision technology.Combining with the trajectory simulation of the isolated model, the reasonably distributed laser stripes and selfluminous markers are utilized to capture clear images of the object.Then, after image processing,feature extraction, stereo correspondence and reconstruction, three-dimensional information of laser stripes and self-luminous markers are calculated.Besides, a pose solution method based on projected laser stripes and self-luminous markers is proposed.Finally, simulation experiments on measuring the position and attitude of high-speed rolling targets are conducted, as well as accuracy verification experiments.Experimental results indicate that the proposed method is feasible and efficient for measuring the pose parameters of rolling targets in wind tunnels.
文摘This paper theoretically analyzes and researches the coordinate frames of a 3D vision scanning system, establishes the mathematic model of a system scanning process, derives the relationship between the general non-orthonormal sensor coordinate system and the machine coordinate system and the coordinate transformation matrix of the extrinsic calibration for the system.
基金supported by Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education(No.2020R1I1A3068274),Received by Junho Ahn.https://www.nrf.re.kr/supported by the Korea Agency for Infrastructure Technology Advancement(KAIA)by the Ministry of Land,Infrastructure and Transport under Grant(No.22QPWO-C152223-04),Received by Chulsu Kim.https://www.kaia.re.kr/.
文摘Existingfirefighting robots are focused on simple storage orfire sup-pression outside buildings rather than detection or recognition.Utilizing a large number of robots using expensive equipment is challenging.This study aims to increase the efficiency of search and rescue operations and the safety offirefigh-ters by detecting and identifying the disaster site by recognizing collapsed areas,obstacles,and rescuers on-site.A fusion algorithm combining a camera and three-dimension light detection and ranging(3D LiDAR)is proposed to detect and loca-lize the interiors of disaster sites.The algorithm detects obstacles by analyzingfloor segmentation and edge patterns using a mask regional convolutional neural network(mask R-CNN)features model based on the visual data collected from a parallelly connected camera and 3D LiDAR.People as objects are detected using you only look once version 4(YOLOv4)in the image data to localize persons requiring rescue.The point cloud data based on 3D LiDAR cluster the objects using the density-based spatial clustering of applications with noise(DBSCAN)clustering algorithm and estimate the distance to the actual object using the center point of the clustering result.The proposed artificial intelligence(AI)algorithm was verified based on individual sensors using a sensor-mounted robot in an actual building to detectfloor surfaces,atypical obstacles,and persons requiring rescue.Accordingly,the fused AI algorithm was comparatively verified.
