Most sensors or cameras discussed in the sensor network community are usually 3D homogeneous, even though their2 D coverage areas in the ground plane are heterogeneous. Meanwhile, observed objects of camera networks a...Most sensors or cameras discussed in the sensor network community are usually 3D homogeneous, even though their2 D coverage areas in the ground plane are heterogeneous. Meanwhile, observed objects of camera networks are usually simplified as 2D points in previous literature. However in actual application scenes, not only cameras are always heterogeneous with different height and action radiuses, but also the observed objects are with 3D features（i.e., height）. This paper presents a sensor planning formulation addressing the efficiency enhancement of visual tracking in 3D heterogeneous camera networks that track and detect people traversing a region. The problem of sensor planning consists of three issues：（i） how to model the 3D heterogeneous cameras;（ii） how to rank the visibility, which ensures that the object of interest is visible in a camera＇s field of view;（iii） how to reconfigure the 3D viewing orientations of the cameras. This paper studies the geometric properties of 3D heterogeneous camera networks and addresses an evaluation formulation to rank the visibility of observed objects. Then a sensor planning method is proposed to improve the efficiency of visual tracking. Finally, the numerical results show that the proposed method can improve the tracking performance of the system compared to the conventional strategies.展开更多
视觉导航作为移动机器人自主运行的核心技术支撑,其性能直接决定移动机器人的环境感知精度、定位建图可靠性与路径规划的合理性。文章系统综述移动机器人视觉导航的研究进展,围绕视觉传感器、同步定位与地图构建(Simultaneous Localizat...视觉导航作为移动机器人自主运行的核心技术支撑,其性能直接决定移动机器人的环境感知精度、定位建图可靠性与路径规划的合理性。文章系统综述移动机器人视觉导航的研究进展,围绕视觉传感器、同步定位与地图构建(Simultaneous Localization and Mapping,SLAM)和路径规划三大核心环节展开分析:在视觉传感器层面,重点探讨单模态、多模态融合视觉传感器和新型视觉传感器的技术特性与适配场景;在SLAM层面,总结传统几何SLAM、多模态融合SLAM以及神经隐式SLAM的技术演进与性能优势;在路径规划层面,重点介绍传统算法与生物启发算法的特点与适用场景。最后,总结当前技术面临的挑战,并对未来研究方向进行展望,为视觉导航技术的进一步发展提供参考。展开更多
Three-dimensional(3 D) visual tracking of a multicopter(where the camera is fixed while the multicopter is moving) means continuously recovering the six-degree-of-freedom pose of the multicopter relative to the camera...Three-dimensional(3 D) visual tracking of a multicopter(where the camera is fixed while the multicopter is moving) means continuously recovering the six-degree-of-freedom pose of the multicopter relative to the camera. It can be used in many applications,such as precision terminal guidance and control algorithm validation for multicopters. However, it is difficult for many researchers to build a 3 D visual tracking system for multicopters(VTSMs) by using cheap and off-the-shelf cameras. This paper firstly gives an overview of the three key technologies of a 3 D VTSMs: multi-camera placement, multi-camera calibration and pose estimation for multicopters. Then, some representative 3 D visual tracking systems for multicopters are introduced. Finally, the future development of the 3D VTSMs is analyzed and summarized.展开更多
In this paper, we propose a method to select the observation position in visual servoing with an eye-in-vehicle configuration for the manipulator. In traditional visual servoing, the images taken by the camera may hav...In this paper, we propose a method to select the observation position in visual servoing with an eye-in-vehicle configuration for the manipulator. In traditional visual servoing, the images taken by the camera may have various problems, including being out of view, large perspective aberrance, improper projection area of object in images and so on. In this paper, we propose a method to determine the observation position to solve these problems. A mobile robot system with pan-tilt camera is designed, which calculates the observation position based on an observation and then moves there. Both simulation and experimental results are provided to validate the effectiveness of the proposed method.展开更多
A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning...A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning level, a motion control level and a servo control level. The last three levels form a local real-time open robot controller, which realizes motion planning and motion control of a robot. A camera calibration method based on the relative movement of the end-effector connected to a robot is proposed and a method for tracking weld seam based on the structured light stereovision is provided. Combining the parameters of the cameras and laser plane, three groups of position values in Cartesian space are obtained for each feature point in a stripe projected on the weld seam. The accurate three-dimensional position of the edge points in the weld seam can be calculated from the obtained parameters with an information fusion algorithm. By calculating the weld seam parameter from position and image data, the movement parameters of the robot used for tracking can be determined. A swing welding experiment of type V groove weld is successfully conducted, the results of which show that the system has high resolution seam tracking in real-time, and works stably and efficiently.展开更多
This paper addresses the robust visual tracking of multi-feature points for a 3D manipulator with unknown intrinsic and extrinsic parameters of the vision system. This class of control systems are highly nonlinear con...This paper addresses the robust visual tracking of multi-feature points for a 3D manipulator with unknown intrinsic and extrinsic parameters of the vision system. This class of control systems are highly nonlinear control systems characterized as time-varying and strong coupling in states and unknown parameters. It is first pointed out that not only is the Jacobian image matrix nonsingular, but also its minimum singular value has a positive limit. This provides the foundation of kinematics and dynamics control of manipulators with visual feedback. Second, the Euler angle expressed rotation transformation is employed to estimate a subspace of the parameter space of the vision system. Based on the two results above, and arbitrarily chosen parameters in this subspace, the tracking controllers are proposed so that the image errors can be made as small as desired so long as the control gain is allowed to be large. The controller does not use visual velocity to achieve high and robust performance with low sampling rate of the vision system. The obtained results are proved by Lyapunov direct method. Experiments are included to demonstrate the effectiveness of the proposed controller.展开更多
In this paper, we proposed an optimized real-time hybrid cooperative multi-camera tracking system for large-scale au-tomate surveillance based on embedded smart cameras including stationary cameras and moving pan/tilt...In this paper, we proposed an optimized real-time hybrid cooperative multi-camera tracking system for large-scale au-tomate surveillance based on embedded smart cameras including stationary cameras and moving pan/tilt/zoom (PTZ) cameras embedded with TI DSP TMS320DM6446 for intelligent visual analysis. Firstly, the overlapping areas and projection relations between adjacent cameras' field of view (FOV) is calculated. Based on the relations of FOV ob-tained and tracking information of each single camera, a homography based target handover procedure is done for long-term multi-camera tracking. After that, we fully implemented the tracking system on the embedded platform de-veloped by our group. Finally, to reduce the huge computational complexity, a novel hierarchical optimization method is proposed. Experimental results demonstrate the robustness and real-time efficiency in dynamic real-world environ-ments and the computational burden is significantly reduced by 98.84%. Our results demonstrate that our proposed sys-tem is capable of tracking targets effectively and achieve large-scale surveillance with clear detailed close-up visual features capturing and recording in dynamic real-life environments.展开更多
为减少因船舶偏离航道而造成的搁浅、碰撞航标或桥墩等水上交通事故,提出了一种基于多目相机自动识别航道的桥区航行异常船舶预警方法。基于YOLOv5(You Only Look Once version 5)目标检测算法,联动变、定焦相机识别并定位航标和船舶,...为减少因船舶偏离航道而造成的搁浅、碰撞航标或桥墩等水上交通事故,提出了一种基于多目相机自动识别航道的桥区航行异常船舶预警方法。基于YOLOv5(You Only Look Once version 5)目标检测算法,联动变、定焦相机识别并定位航标和船舶,跟踪并记录船舶航迹点,计算船舶的速度和航向并推算船位。提出了一种基于视频船舶航迹点的密度聚类识别航道两侧航标的方法,实现航道自适应可视化。基于船位推算识别并预警航行状态异常的船舶。实验结果表明:航标、船舶的检测正确率分别达84.8%、90.3%,相较单一相机检测模型,正确率分别提高了32.1%、5.5%;能够自适应可视化航道并识别、预警航行异常船舶。展开更多
基金supported by the National Natural Science Foundationof China(61100207)the National Key Technology Research and Development Program of the Ministry of Science and Technology of China(2014BAK14B03)+1 种基金the Fundamental Research Funds for the Central Universities(2013PT132013XZ12)
文摘Most sensors or cameras discussed in the sensor network community are usually 3D homogeneous, even though their2 D coverage areas in the ground plane are heterogeneous. Meanwhile, observed objects of camera networks are usually simplified as 2D points in previous literature. However in actual application scenes, not only cameras are always heterogeneous with different height and action radiuses, but also the observed objects are with 3D features（i.e., height）. This paper presents a sensor planning formulation addressing the efficiency enhancement of visual tracking in 3D heterogeneous camera networks that track and detect people traversing a region. The problem of sensor planning consists of three issues：（i） how to model the 3D heterogeneous cameras;（ii） how to rank the visibility, which ensures that the object of interest is visible in a camera＇s field of view;（iii） how to reconfigure the 3D viewing orientations of the cameras. This paper studies the geometric properties of 3D heterogeneous camera networks and addresses an evaluation formulation to rank the visibility of observed objects. Then a sensor planning method is proposed to improve the efficiency of visual tracking. Finally, the numerical results show that the proposed method can improve the tracking performance of the system compared to the conventional strategies.
文摘视觉导航作为移动机器人自主运行的核心技术支撑,其性能直接决定移动机器人的环境感知精度、定位建图可靠性与路径规划的合理性。文章系统综述移动机器人视觉导航的研究进展,围绕视觉传感器、同步定位与地图构建(Simultaneous Localization and Mapping,SLAM)和路径规划三大核心环节展开分析:在视觉传感器层面,重点探讨单模态、多模态融合视觉传感器和新型视觉传感器的技术特性与适配场景;在SLAM层面,总结传统几何SLAM、多模态融合SLAM以及神经隐式SLAM的技术演进与性能优势;在路径规划层面,重点介绍传统算法与生物启发算法的特点与适用场景。最后,总结当前技术面临的挑战,并对未来研究方向进行展望,为视觉导航技术的进一步发展提供参考。
基金supported by the National Key Research and Development Program of China (No. 2017YFB1300102)National Natural Science Foundation of China (No. 61803025)
文摘Three-dimensional(3 D) visual tracking of a multicopter(where the camera is fixed while the multicopter is moving) means continuously recovering the six-degree-of-freedom pose of the multicopter relative to the camera. It can be used in many applications,such as precision terminal guidance and control algorithm validation for multicopters. However, it is difficult for many researchers to build a 3 D visual tracking system for multicopters(VTSMs) by using cheap and off-the-shelf cameras. This paper firstly gives an overview of the three key technologies of a 3 D VTSMs: multi-camera placement, multi-camera calibration and pose estimation for multicopters. Then, some representative 3 D visual tracking systems for multicopters are introduced. Finally, the future development of the 3D VTSMs is analyzed and summarized.
基金supported by Natural Science Foundation of China (No. 61773374)Key Research and Development Program of China (No. 2017YFB1300104)
文摘In this paper, we propose a method to select the observation position in visual servoing with an eye-in-vehicle configuration for the manipulator. In traditional visual servoing, the images taken by the camera may have various problems, including being out of view, large perspective aberrance, improper projection area of object in images and so on. In this paper, we propose a method to determine the observation position to solve these problems. A mobile robot system with pan-tilt camera is designed, which calculates the observation position based on an observation and then moves there. Both simulation and experimental results are provided to validate the effectiveness of the proposed method.
基金Supported by National Natural Science Foundation of China（60873032, 61105095, 61203361） Doctoral Program Foundation of Ministry of Education of China （20100142110020）+3 种基金 the Specialized Research Fund for the Doctoral Program of Higher Education of China （20100073120020） Postdoctoral Science Foundation of China （2012M511095） Shanghai Municipal Natural Science Foundation （11ZR1418400） Shanghai Postdoctoral Science Foundation（12R21414200）
基金This work was supported by the National High Technology Research and Development Program of China under Grant 2002AA422160 by the National Key Fundamental Research and the Devel-opment Project of China (973) under Grant 2002CB312200.
文摘A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning level, a motion control level and a servo control level. The last three levels form a local real-time open robot controller, which realizes motion planning and motion control of a robot. A camera calibration method based on the relative movement of the end-effector connected to a robot is proposed and a method for tracking weld seam based on the structured light stereovision is provided. Combining the parameters of the cameras and laser plane, three groups of position values in Cartesian space are obtained for each feature point in a stripe projected on the weld seam. The accurate three-dimensional position of the edge points in the weld seam can be calculated from the obtained parameters with an information fusion algorithm. By calculating the weld seam parameter from position and image data, the movement parameters of the robot used for tracking can be determined. A swing welding experiment of type V groove weld is successfully conducted, the results of which show that the system has high resolution seam tracking in real-time, and works stably and efficiently.
基金This work was supported by The National Science Foundation(No.60474009),Shu Guang Program(No.05SG48)Scientific Programm ofShanghai Education Committee(No.07zz90).
文摘This paper addresses the robust visual tracking of multi-feature points for a 3D manipulator with unknown intrinsic and extrinsic parameters of the vision system. This class of control systems are highly nonlinear control systems characterized as time-varying and strong coupling in states and unknown parameters. It is first pointed out that not only is the Jacobian image matrix nonsingular, but also its minimum singular value has a positive limit. This provides the foundation of kinematics and dynamics control of manipulators with visual feedback. Second, the Euler angle expressed rotation transformation is employed to estimate a subspace of the parameter space of the vision system. Based on the two results above, and arbitrarily chosen parameters in this subspace, the tracking controllers are proposed so that the image errors can be made as small as desired so long as the control gain is allowed to be large. The controller does not use visual velocity to achieve high and robust performance with low sampling rate of the vision system. The obtained results are proved by Lyapunov direct method. Experiments are included to demonstrate the effectiveness of the proposed controller.
文摘In this paper, we proposed an optimized real-time hybrid cooperative multi-camera tracking system for large-scale au-tomate surveillance based on embedded smart cameras including stationary cameras and moving pan/tilt/zoom (PTZ) cameras embedded with TI DSP TMS320DM6446 for intelligent visual analysis. Firstly, the overlapping areas and projection relations between adjacent cameras' field of view (FOV) is calculated. Based on the relations of FOV ob-tained and tracking information of each single camera, a homography based target handover procedure is done for long-term multi-camera tracking. After that, we fully implemented the tracking system on the embedded platform de-veloped by our group. Finally, to reduce the huge computational complexity, a novel hierarchical optimization method is proposed. Experimental results demonstrate the robustness and real-time efficiency in dynamic real-world environ-ments and the computational burden is significantly reduced by 98.84%. Our results demonstrate that our proposed sys-tem is capable of tracking targets effectively and achieve large-scale surveillance with clear detailed close-up visual features capturing and recording in dynamic real-life environments.
文摘为减少因船舶偏离航道而造成的搁浅、碰撞航标或桥墩等水上交通事故,提出了一种基于多目相机自动识别航道的桥区航行异常船舶预警方法。基于YOLOv5(You Only Look Once version 5)目标检测算法,联动变、定焦相机识别并定位航标和船舶,跟踪并记录船舶航迹点,计算船舶的速度和航向并推算船位。提出了一种基于视频船舶航迹点的密度聚类识别航道两侧航标的方法,实现航道自适应可视化。基于船位推算识别并预警航行状态异常的船舶。实验结果表明:航标、船舶的检测正确率分别达84.8%、90.3%,相较单一相机检测模型,正确率分别提高了32.1%、5.5%;能够自适应可视化航道并识别、预警航行异常船舶。
