Visual localization is a crucial component in the application of mobile robot and autonomous driving.Image retrieval is an efficient and effective technique in image-based localization methods.Due to the drastic varia...Visual localization is a crucial component in the application of mobile robot and autonomous driving.Image retrieval is an efficient and effective technique in image-based localization methods.Due to the drastic variability of environmental conditions,e.g.,illumination changes,retrievalbased visual localization is severely affected and becomes a challenging problem.In this work,a general architecture is first formulated probabilistically to extract domain-invariant features through multi-domain image translation.Then,a novel gradientweighted similarity activation mapping loss(Grad-SAM)is incorporated for finer localization with high accuracy.We also propose a new adaptive triplet loss to boost the contrastive learning of the embedding in a self-supervised manner.The final coarse-to-fine image retrieval pipeline is implemented as the sequential combination of models with and without Grad-SAM loss.Extensive experiments have been conducted to validate the effectiveness of the proposed approach on the CMU-Seasons dataset.The strong generalization ability of our approach is verified with the RobotCar dataset using models pre-trained on urban parts of the CMU-Seasons dataset.Our performance is on par with or even outperforms the state-of-the-art image-based localization baselines in medium or high precision,especially under challenging environments with illumination variance,vegetation,and night-time images.Moreover,real-site experiments have been conducted to validate the efficiency and effectiveness of the coarse-to-fine strategy for localization.展开更多
When a vehicle travels in urban areas, onboard global positioning system （GPS） signals may be obstructed by high-rise buildings and thereby cannot provide accurate positions. It is proposed to perform localization b...When a vehicle travels in urban areas, onboard global positioning system （GPS） signals may be obstructed by high-rise buildings and thereby cannot provide accurate positions. It is proposed to perform localization by registering ground images to a 2D building boundary map which is generated from aerial images. Multilayer feature graphs （MFG） is employed to model building facades from the ground images. MFG was reported in the previous work to facilitate the robot scene understand- ing in urhan areas. By constructing MFG, the 2D/3D positions of features can be obtained, inclu- cling line segments, ideal lines, and all primary vertical planes. Finally, a voting-based feature weighted localization method is developed based on MFGs and the 2D building boundary map. The proposed method has been implemented and validated in physical experiments. In the proposed ex- periments, the algorithm has achieved an overall localization accuracy of 2.2m, which is better than commercial GPS working in open environments.展开更多
In feature-based visual localization for small-scale scenes,local descriptors are used to estimate the camera pose of a query image.For large and ambiguous environments,learning-based hierarchical networks that employ...In feature-based visual localization for small-scale scenes,local descriptors are used to estimate the camera pose of a query image.For large and ambiguous environments,learning-based hierarchical networks that employ local as well as global descriptors to reduce the search space of database images into a smaller set of reference views have been introduced.However,since global descriptors are generated using visual features,reference images with some of these features may be erroneously selected.In order to address this limitation,this paper proposes two clustering methods based on how often features appear as well as their covisibility.For both approaches,the scene is represented by voxels whose size and number are computed according to the size of the scene and the number of available 3Dpoints.In the first approach,a voxel-based histogram representing highly reoccurring scene regions is generated from reference images.A meanshift is then employed to group the most highly reoccurring voxels into place clusters based on their spatial proximity.In the second approach,a graph representing the covisibility-based relationship of voxels is built.Local matching is performed within the reference image clusters,and a perspective-n-point is employed to estimate the camera pose.The experimental results showed that camera pose estimation using the proposed approaches was more accurate than that of previous methods.展开更多
A localization method based on distance function of projected features is presented to solve the accuracy reduction or failure problem due to occlusion and blurring caused by smog, when dealing with vision based local...A localization method based on distance function of projected features is presented to solve the accuracy reduction or failure problem due to occlusion and blurring caused by smog, when dealing with vision based localization for target oil and gas wellhead （OGWH）. Firstly, the target OGWH is modeled as a cylinder with marker, and a vector with redundant parameter is used to describe its pose. Secondly, the explicit mapping relationship between the pose vector with redundant parameter and projected features is derived. Then, a 2D-point-to-feature distance function is proposed, as well as its derivative. Finally, based on this distance function and its derivative, an algorithm is proposed to estimate the pose of target OGWH directly according to the 2D image information, and the validity of the method is verified by both synthetic data and real image experiments. The results show that this method is able to accomplish the localization in the case of occlusion and blurring, and its anti-noise ability is good especially with noise ratio of less than 70%.展开更多
Localization along fixed routes is the fundamental function of transportation applications,including patrol vehicles,shuttles,buses,and even passenger vehicles.To achieve accurate and reliable localization,we propose ...Localization along fixed routes is the fundamental function of transportation applications,including patrol vehicles,shuttles,buses,and even passenger vehicles.To achieve accurate and reliable localization,we propose a tightly coupled A Priori Map Simultaneous Localization and Mapping(APM-SLAM)system.APM-SLAM provides a comprehensive and heterogeneous framework,encompassing both mapping and localization processes.The mapping stage leverages Global Navigation Satellite System(GNSS)-aided Structure from Motion(SfM)to establish reliable a priori maps with coarse-and fine-level components.The localization process integrates coarse-to-fine matching with Maximum A Posteriori(MAP)Probability estimation to refine pose accuracy.By incorporating deep learning-based features and point descriptors,our system maintains robustness even in scenarios with significant visual variation.Unlike traditional map-based approaches,APM-SLAM models the a priori map’s point structures as probabilistic distributions and incorporates them into the optimization process.Extensive experiments on public datasets demonstrate the superiority of our method in both mapping precision and localization accuracy,achieving decimeter-level translation precision.Ablation studies further validate the effectiveness of each component within our system.This work contributes to establishing maps and utilizing a priori information for localization simultaneously.展开更多
Indoor visual localization,i.e.,6 Degree-of-Freedom camera pose estimation for a query image with respect to a known scene,is gaining increased attention driven by rapid progress of applications such as robotics and a...Indoor visual localization,i.e.,6 Degree-of-Freedom camera pose estimation for a query image with respect to a known scene,is gaining increased attention driven by rapid progress of applications such as robotics and augmented reality.However,drastic visual discrepancies between an onsite query image and prerecorded indoor images cast a significant challenge for visual localization.In this paper,based on the key observation of the constant existence of planar surfaces such as floors or walls in indoor scenes,we propose a novel system incorporating geometric information to address issues using only pixelated images.Through the system implementation,we contribute a hierarchical structure consisting of pre-scanned images and point cloud,as well as a distilled representation of the planar-element layout extracted from the original dataset.A view synthesis procedure is designed to generate synthetic images as complementary to that of a sparsely sampled dataset.Moreover,a global image descriptor based on the image statistic modality,called block mean,variance,and color(BMVC),was employed to speed up the candidate pose identification incorporated with a traditional convolutional neural network(CNN)descriptor.Experimental results on a popular benchmark demonstrate that the proposed method outperforms the state-of-the-art approaches in terms of visual localization validity and accuracy.展开更多
To overcome the shortcomings of existing robot localization sensors, such as low accuracy and poor robustness, a high precision visual localization system based on infrared-reflective artificial markers is designed an...To overcome the shortcomings of existing robot localization sensors, such as low accuracy and poor robustness, a high precision visual localization system based on infrared-reflective artificial markers is designed and illustrated in detail in this paper. First, the hardware system of the localization sensor is developed. Secondly, we design a novel kind of infrared-reflective artificial marker whose characteristics can be extracted by the acquisition and processing of the infrared image. In addition, a confidence calculation method for marker identification is proposed to obtain the probabilistic localization results. Finally, the autonomous localization of the robot is achieved by calculating the relative pose relation between the robot and the artificial marker based on the perspective-3-point(P3P) visual localization algorithm. Numerous experiments and practical applications show that the designed localization sensor system is immune to the interferences of the illumination and observation angle changes. The precision of the sensor is ±1.94 cm for position localization and ±1.64° for angle localization. Therefore, it satisfies perfectly the requirements of localization precision for an indoor mobile robot.展开更多
With the rapid popularization of mobile devices and the wide application of various sensors,scene perception methods applied to mobile devices occupy an important position in location-based services such as navigation...With the rapid popularization of mobile devices and the wide application of various sensors,scene perception methods applied to mobile devices occupy an important position in location-based services such as navigation and augmented reality(AR).The development of deep learning technologies has greatly improved the visual perception ability of machines to scenes.The basic framework of scene visual perception,related technologies and the specific process applied to AR navigation are introduced,and future technology development is proposed.An application(APP)is designed to improve the application effect of AR navigation.The APP includes three modules:navigation map generation,cloud navigation algorithm,and client design.The navigation map generation tool works offline.The cloud saves the navigation map and provides navigation algorithms for the terminal.The terminal realizes local real-time positioning and AR path rendering.展开更多
This paper presents a hierarchical simultaneous localization and mapping（SLAM） system for a small unmanned aerial vehicle（UAV） using the output of an inertial measurement unit（IMU） and the bearing-only observati...This paper presents a hierarchical simultaneous localization and mapping（SLAM） system for a small unmanned aerial vehicle（UAV） using the output of an inertial measurement unit（IMU） and the bearing-only observations from an onboard monocular camera.A homography based approach is used to calculate the motion of the vehicle in 6 degrees of freedom by image feature match.This visual measurement is fused with the inertial outputs by an indirect extended Kalman filter（EKF） for attitude and velocity estimation.Then,another EKF is employed to estimate the position of the vehicle and the locations of the features in the map.Both simulations and experiments are carried out to test the performance of the proposed system.The result of the comparison with the referential global positioning system/inertial navigation system（GPS/INS） navigation indicates that the proposed SLAM can provide reliable and stable state estimation for small UAVs in GPS-denied environments.展开更多
In this paper,we present a novel algorithm for odometry estimation based on ceiling vision.The main contribution of this algorithm is the introduction of principal direction detection that can greatly reduce error acc...In this paper,we present a novel algorithm for odometry estimation based on ceiling vision.The main contribution of this algorithm is the introduction of principal direction detection that can greatly reduce error accumulation problem in most visual odometry estimation approaches.The principal direction is defned based on the fact that our ceiling is flled with artifcial vertical and horizontal lines which can be used as reference for the current robot s heading direction.The proposed approach can be operated in real-time and it performs well even with camera s disturbance.A moving low-cost RGB-D camera（Kinect）,mounted on a robot,is used to continuously acquire point clouds.Iterative closest point（ICP） is the common way to estimate the current camera position by registering the currently captured point cloud to the previous one.However,its performance sufers from data association problem or it requires pre-alignment information.The performance of the proposed principal direction detection approach does not rely on data association knowledge.Using this method,two point clouds are properly pre-aligned.Hence,we can use ICP to fne-tune the transformation parameters and minimize registration error.Experimental results demonstrate the performance and stability of the proposed system under disturbance in real-time.Several indoor tests are carried out to show that the proposed visual odometry estimation method can help to signifcantly improve the accuracy of simultaneous localization and mapping（SLAM）.展开更多
Purified terephthalic acid(PTA) is an important chemical raw material. P-xylene(PX) is transformed to terephthalic acid(TA) through oxidation process and TA is refined to produce PTA. The PX oxidation reaction is a co...Purified terephthalic acid(PTA) is an important chemical raw material. P-xylene(PX) is transformed to terephthalic acid(TA) through oxidation process and TA is refined to produce PTA. The PX oxidation reaction is a complex process involving three-phase reaction of gas, liquid and solid. To monitor the process and to improve the product quality, as well as to visualize the fault type clearly, a fault diagnosis method based on selforganizing map(SOM) and high dimensional feature extraction method, local tangent space alignment(LTSA),is proposed. In this method, LTSA can reduce the dimension and keep the topology information simultaneously,and SOM distinguishes various states on the output map. Monitoring results of PX oxidation reaction process indicate that the LTSA–SOM can well detect and visualize the fault type.展开更多
Visual localization and object detection both play important roles in various tasks.In many indoor application scenarios where some detected objects have fixed positions,the two techniques work closely together.Howeve...Visual localization and object detection both play important roles in various tasks.In many indoor application scenarios where some detected objects have fixed positions,the two techniques work closely together.However,few researchers consider these two tasks simultaneously,because of a lack of datasets and the little attention paid to such environments.In this paper,we explore multi-task network design and joint refinement of detection and localization.To address the dataset problem,we construct a medium indoor scene of an aviation exhibition hall through a semi-automatic process.The dataset provides localization and detection information,and is publicly available at https://drive.google.com/drive/folders/1U28zk0N4_I0db zkqyIAK1A15k9oUKOjI?usp=sharing for benchmarking localization and object detection tasks.Targeting this dataset,we have designed a multi-task network,JLDNet,based on YOLO v3,that outputs a target point cloud and object bounding boxes.For dynamic environments,the detection branch also promotes the perception of dynamics.JLDNet includes image feature learning,point feature learning,feature fusion,detection construction,and point cloud regression.Moreover,object-level bundle adjustment is used to further improve localization and detection accuracy.To test JLDNet and compare it to other methods,we have conducted experiments on 7 static scenes,our constructed dataset,and the dynamic TUM RGB-D and Bonn datasets.Our results show state-of-the-art accuracy for both tasks,and the benefit of jointly working on both tasks is demonstrated.展开更多
To solve the problem of mobile robots needing to adjust their pose for accurate operation after reaching the target point in the indoor environment,a localization method based on scene modeling and recognition has bee...To solve the problem of mobile robots needing to adjust their pose for accurate operation after reaching the target point in the indoor environment,a localization method based on scene modeling and recognition has been designed.Firstly,the offline scene model is created by both handcrafted feature and semantic feature.Then,the scene recognition and location calculation are performed online based on the offline scene model.To improve the accuracy of recognition and location calculation,this paper proposes a method that integrates both semantic features matching and handcrafted features matching.Based on the results of scene recognition,the accurate location is obtained through metric calculation with 3D information.The experimental results show that the accuracy of scene recognition is over 90%,and the average localization error is less than 1 meter.Experimental results demonstrate that the localization has a better performance after using the proposed improved method.展开更多
A new visual measurement method is proposed to estimate three-dimensional （3D） position of the object on the floor based on a single camera. The camera fixed on a robot is in an inclined position with respect to the...A new visual measurement method is proposed to estimate three-dimensional （3D） position of the object on the floor based on a single camera. The camera fixed on a robot is in an inclined position with respect to the floor. A measurement model with the camera＇s extrinsic parameters such as the height and pitch angle is described. Single image of a chessboard pattern placed on the floor is enough to calibrate the camera＇s extrinsic parameters after the camera＇s intrinsic parameters are calibrated. Then the position of object on the floor can be computed with the measurement model. Furthermore, the height of object can be calculated with the paired-points in the vertical line sharing the same position on the floor. Compared to the conventional method used to estimate the positions on the plane, this method can obtain the 3D positions. The indoor experiment testifies the accuracy and validity of the proposed method.展开更多
Simultaneous Localization and Mapping(SLAM)has been widely used in emergency response,self-driving and city-scale 3D mapping and navigation.Recent deep-learning based feature point extractors have demonstrated superio...Simultaneous Localization and Mapping(SLAM)has been widely used in emergency response,self-driving and city-scale 3D mapping and navigation.Recent deep-learning based feature point extractors have demonstrated superior performance in dealing with the complex environmental challenges(e.g.extreme lighting)while the traditional extractors are struggling.In this paper,we have successfully improved the robustness and accuracy of a monocular visual SLAM system under various complex scenes by adding a deep learning based visual localization thread as an augmentation to the visual SLAM framework.In this thread,our feature extractor with an efficient lightweight deep neural network is used for absolute pose and scale estimation in real time using the highly accurate georeferenced prior map database at 20cm geometric accuracy created by our in-house and low-cost LiDAR and camera integrated device.The closed-loop error provided by our SLAM system with and without this enhancement is 1.03m and 18.28m respectively.The scale estimation of the monocular visual SLAM is also significantly improved(0.01 versus 0.98).In addition,a novel camera-LiDAR calibration workflow is also provided for large-scale 3D mapping.This paper demonstrates the application and research potential of deep-learning based vision SLAM with image and LiDAR sensors.展开更多
Sensitivity analysis is a powerful method for discovering the significant factors that contribute to understanding the interaction between variables in multivariate datasets. A number of sensitivity analysis methods f...Sensitivity analysis is a powerful method for discovering the significant factors that contribute to understanding the interaction between variables in multivariate datasets. A number of sensitivity analysis methods fall into the class of local analysis, in which the sensitivity is defined as the partial derivatives of a target variable with respect to a group of independent variables. In a recent paper, we presented a novel pointwise local pattern exploration system for visual sensitivity analysis. Using this system, analysts are able to explore local patterns and the sensitivity at individual data points, which reveals the relationships between a focal point and its neighbors this paper we present several evaluations of the system, including case studies with real datasets, user studies the effectiveness of the visualizations and interactions, and a detailed description of the experience of a user In on展开更多
The extraction and description of image features are very important for visual simultaneous localization and mapping(V-SLAM).A rotated boosted efficient binary local image descriptor(BEBLID)SLAM(RB-SLAM)algorithm base...The extraction and description of image features are very important for visual simultaneous localization and mapping(V-SLAM).A rotated boosted efficient binary local image descriptor(BEBLID)SLAM(RB-SLAM)algorithm based on improved oriented fast and rotated brief(ORB)feature description is proposed in this paper,which can solve the problems of low localization accuracy and time efficiency of the current ORB-SLAM3 algorithm.Firstly,it uses the BEBLID to replace the feature point description algorithm of the original ORB to enhance the expressiveness and description efficiency of the image.Secondly,it adds rotational invariance to the BEBLID using the orientation information of the feature points.It also selects the rotationally stable bits in the BEBLID to further enhance the rotational invariance of the BEBLID.Finally,it retrains the binary visual dictionary based on the BEBLID to reduce the cumulative error of V-SLAM and improve the loading speed of the visual dictionary.Experiments show that the dictionary loading efficiency is improved by more than 10 times.The RB-SLAM algorithm improves the trajectory accuracy by 24.75%on the TUM dataset and 26.25%on the EuRoC dataset compared to the ORB-SLAM3 algorithm.展开更多
The cross-view matching of local image features is a fundamental task in visual localization and 3D reconstruction.This study proposes FilterGNN,a transformer-based graph neural network(GNN),aiming to improve the matc...The cross-view matching of local image features is a fundamental task in visual localization and 3D reconstruction.This study proposes FilterGNN,a transformer-based graph neural network(GNN),aiming to improve the matching efficiency and accuracy of visual descriptors.Based on high matching sparseness and coarse-to-fine covisible area detection,FilterGNN utilizes cascaded optimal graph-matching filter modules to dynamically reject outlier matches.Moreover,we successfully adapted linear attention in FilterGNN with post-instance normalization support,which significantly reduces the complexity of complete graph learning from O(N2)to O(N).Experiments show that FilterGNN requires only 6%of the time cost and 33.3%of the memory cost compared with SuperGlue under a large-scale input size and achieves a competitive performance in various tasks,such as pose estimation,visual localization,and sparse 3D reconstruction.展开更多
文摘Visual localization is a crucial component in the application of mobile robot and autonomous driving.Image retrieval is an efficient and effective technique in image-based localization methods.Due to the drastic variability of environmental conditions,e.g.,illumination changes,retrievalbased visual localization is severely affected and becomes a challenging problem.In this work,a general architecture is first formulated probabilistically to extract domain-invariant features through multi-domain image translation.Then,a novel gradientweighted similarity activation mapping loss(Grad-SAM)is incorporated for finer localization with high accuracy.We also propose a new adaptive triplet loss to boost the contrastive learning of the embedding in a self-supervised manner.The final coarse-to-fine image retrieval pipeline is implemented as the sequential combination of models with and without Grad-SAM loss.Extensive experiments have been conducted to validate the effectiveness of the proposed approach on the CMU-Seasons dataset.The strong generalization ability of our approach is verified with the RobotCar dataset using models pre-trained on urban parts of the CMU-Seasons dataset.Our performance is on par with or even outperforms the state-of-the-art image-based localization baselines in medium or high precision,especially under challenging environments with illumination variance,vegetation,and night-time images.Moreover,real-site experiments have been conducted to validate the efficiency and effectiveness of the coarse-to-fine strategy for localization.
基金Supported by the National High Technology Research and Development Program of China(No.2012AA041403)National Natural Science Foundation of China(No.60905061,61305107)+1 种基金the Fundamental Research Funds for the Central Universities(No.ZXH2012N003)the Scientific Research Funds for Civil Aviation University of China(No.2012QD23x)
文摘When a vehicle travels in urban areas, onboard global positioning system （GPS） signals may be obstructed by high-rise buildings and thereby cannot provide accurate positions. It is proposed to perform localization by registering ground images to a 2D building boundary map which is generated from aerial images. Multilayer feature graphs （MFG） is employed to model building facades from the ground images. MFG was reported in the previous work to facilitate the robot scene understand- ing in urhan areas. By constructing MFG, the 2D/3D positions of features can be obtained, inclu- cling line segments, ideal lines, and all primary vertical planes. Finally, a voting-based feature weighted localization method is developed based on MFGs and the 2D building boundary map. The proposed method has been implemented and validated in physical experiments. In the proposed ex- periments, the algorithm has achieved an overall localization accuracy of 2.2m, which is better than commercial GPS working in open environments.
基金supported by the Basic Science Research Program through the National Research Foundation of Korea(NRF)funded by the Ministry of Education(NRF-2018R1D1A1B07049932).
文摘In feature-based visual localization for small-scale scenes,local descriptors are used to estimate the camera pose of a query image.For large and ambiguous environments,learning-based hierarchical networks that employ local as well as global descriptors to reduce the search space of database images into a smaller set of reference views have been introduced.However,since global descriptors are generated using visual features,reference images with some of these features may be erroneously selected.In order to address this limitation,this paper proposes two clustering methods based on how often features appear as well as their covisibility.For both approaches,the scene is represented by voxels whose size and number are computed according to the size of the scene and the number of available 3Dpoints.In the first approach,a voxel-based histogram representing highly reoccurring scene regions is generated from reference images.A meanshift is then employed to group the most highly reoccurring voxels into place clusters based on their spatial proximity.In the second approach,a graph representing the covisibility-based relationship of voxels is built.Local matching is performed within the reference image clusters,and a perspective-n-point is employed to estimate the camera pose.The experimental results showed that camera pose estimation using the proposed approaches was more accurate than that of previous methods.
基金supported by National Natural Science Foundation of China(No.61403226)the State Key Laboratory of Tribology of China(No.SKLT09A03)
文摘A localization method based on distance function of projected features is presented to solve the accuracy reduction or failure problem due to occlusion and blurring caused by smog, when dealing with vision based localization for target oil and gas wellhead （OGWH）. Firstly, the target OGWH is modeled as a cylinder with marker, and a vector with redundant parameter is used to describe its pose. Secondly, the explicit mapping relationship between the pose vector with redundant parameter and projected features is derived. Then, a 2D-point-to-feature distance function is proposed, as well as its derivative. Finally, based on this distance function and its derivative, an algorithm is proposed to estimate the pose of target OGWH directly according to the 2D image information, and the validity of the method is verified by both synthetic data and real image experiments. The results show that this method is able to accomplish the localization in the case of occlusion and blurring, and its anti-noise ability is good especially with noise ratio of less than 70%.
基金supported by the Project from Shenzhen Science and Technology Innovation Commission(KJZD20230923114810022).
文摘Localization along fixed routes is the fundamental function of transportation applications,including patrol vehicles,shuttles,buses,and even passenger vehicles.To achieve accurate and reliable localization,we propose a tightly coupled A Priori Map Simultaneous Localization and Mapping(APM-SLAM)system.APM-SLAM provides a comprehensive and heterogeneous framework,encompassing both mapping and localization processes.The mapping stage leverages Global Navigation Satellite System(GNSS)-aided Structure from Motion(SfM)to establish reliable a priori maps with coarse-and fine-level components.The localization process integrates coarse-to-fine matching with Maximum A Posteriori(MAP)Probability estimation to refine pose accuracy.By incorporating deep learning-based features and point descriptors,our system maintains robustness even in scenarios with significant visual variation.Unlike traditional map-based approaches,APM-SLAM models the a priori map’s point structures as probabilistic distributions and incorporates them into the optimization process.Extensive experiments on public datasets demonstrate the superiority of our method in both mapping precision and localization accuracy,achieving decimeter-level translation precision.Ablation studies further validate the effectiveness of each component within our system.This work contributes to establishing maps and utilizing a priori information for localization simultaneously.
基金supported by the National Natural Science Foundation of China under Grant Nos.62072284 and 61772318the Special Project of Science and Technology Innovation Base of Key Laboratory of Shandong Province for Software Engineering under Grant No.11480004042015。
文摘Indoor visual localization,i.e.,6 Degree-of-Freedom camera pose estimation for a query image with respect to a known scene,is gaining increased attention driven by rapid progress of applications such as robotics and augmented reality.However,drastic visual discrepancies between an onsite query image and prerecorded indoor images cast a significant challenge for visual localization.In this paper,based on the key observation of the constant existence of planar surfaces such as floors or walls in indoor scenes,we propose a novel system incorporating geometric information to address issues using only pixelated images.Through the system implementation,we contribute a hierarchical structure consisting of pre-scanned images and point cloud,as well as a distilled representation of the planar-element layout extracted from the original dataset.A view synthesis procedure is designed to generate synthetic images as complementary to that of a sparsely sampled dataset.Moreover,a global image descriptor based on the image statistic modality,called block mean,variance,and color(BMVC),was employed to speed up the candidate pose identification incorporated with a traditional convolutional neural network(CNN)descriptor.Experimental results on a popular benchmark demonstrate that the proposed method outperforms the state-of-the-art approaches in terms of visual localization validity and accuracy.
基金supported by the National High-Tech R&D Program(863)of China(No.2009AA04Z220)the National Natural Science Foundation of China(No.61375084)the Key Program of Shandong Provincial Natural Science Foundation,China(No.ZR2015QZ08)
文摘To overcome the shortcomings of existing robot localization sensors, such as low accuracy and poor robustness, a high precision visual localization system based on infrared-reflective artificial markers is designed and illustrated in detail in this paper. First, the hardware system of the localization sensor is developed. Secondly, we design a novel kind of infrared-reflective artificial marker whose characteristics can be extracted by the acquisition and processing of the infrared image. In addition, a confidence calculation method for marker identification is proposed to obtain the probabilistic localization results. Finally, the autonomous localization of the robot is achieved by calculating the relative pose relation between the robot and the artificial marker based on the perspective-3-point(P3P) visual localization algorithm. Numerous experiments and practical applications show that the designed localization sensor system is immune to the interferences of the illumination and observation angle changes. The precision of the sensor is ±1.94 cm for position localization and ±1.64° for angle localization. Therefore, it satisfies perfectly the requirements of localization precision for an indoor mobile robot.
基金supported by ZTE Industry⁃University⁃Institute Coopera⁃tion Funds under Grant No.HC⁃CN⁃20210707004.
文摘With the rapid popularization of mobile devices and the wide application of various sensors,scene perception methods applied to mobile devices occupy an important position in location-based services such as navigation and augmented reality(AR).The development of deep learning technologies has greatly improved the visual perception ability of machines to scenes.The basic framework of scene visual perception,related technologies and the specific process applied to AR navigation are introduced,and future technology development is proposed.An application(APP)is designed to improve the application effect of AR navigation.The APP includes three modules:navigation map generation,cloud navigation algorithm,and client design.The navigation map generation tool works offline.The cloud saves the navigation map and provides navigation algorithms for the terminal.The terminal realizes local real-time positioning and AR path rendering.
基金supported by National High Technology Research Development Program of China (863 Program) (No.2011AA040202)National Science Foundation of China (No.51005008)
文摘This paper presents a hierarchical simultaneous localization and mapping（SLAM） system for a small unmanned aerial vehicle（UAV） using the output of an inertial measurement unit（IMU） and the bearing-only observations from an onboard monocular camera.A homography based approach is used to calculate the motion of the vehicle in 6 degrees of freedom by image feature match.This visual measurement is fused with the inertial outputs by an indirect extended Kalman filter（EKF） for attitude and velocity estimation.Then,another EKF is employed to estimate the position of the vehicle and the locations of the features in the map.Both simulations and experiments are carried out to test the performance of the proposed system.The result of the comparison with the referential global positioning system/inertial navigation system（GPS/INS） navigation indicates that the proposed SLAM can provide reliable and stable state estimation for small UAVs in GPS-denied environments.
文摘In this paper,we present a novel algorithm for odometry estimation based on ceiling vision.The main contribution of this algorithm is the introduction of principal direction detection that can greatly reduce error accumulation problem in most visual odometry estimation approaches.The principal direction is defned based on the fact that our ceiling is flled with artifcial vertical and horizontal lines which can be used as reference for the current robot s heading direction.The proposed approach can be operated in real-time and it performs well even with camera s disturbance.A moving low-cost RGB-D camera（Kinect）,mounted on a robot,is used to continuously acquire point clouds.Iterative closest point（ICP） is the common way to estimate the current camera position by registering the currently captured point cloud to the previous one.However,its performance sufers from data association problem or it requires pre-alignment information.The performance of the proposed principal direction detection approach does not rely on data association knowledge.Using this method,two point clouds are properly pre-aligned.Hence,we can use ICP to fne-tune the transformation parameters and minimize registration error.Experimental results demonstrate the performance and stability of the proposed system under disturbance in real-time.Several indoor tests are carried out to show that the proposed visual odometry estimation method can help to signifcantly improve the accuracy of simultaneous localization and mapping（SLAM）.
基金Supported by the Major State Basic Research Development Program of China(2012CB720500)the National Natural Science Foundation of China(6133301021276078)+3 种基金the National Science Fund for Outstanding Young Scholars(61222303)the Fundamental Research Funds for the Central Universities,Shanghai Rising-Star Program(13QH1401200)the Program for New Century Excellent Talents in University(NCET-10-0885)Shanghai R&D Platform Construction Program(13DZ2295300)
文摘Purified terephthalic acid(PTA) is an important chemical raw material. P-xylene(PX) is transformed to terephthalic acid(TA) through oxidation process and TA is refined to produce PTA. The PX oxidation reaction is a complex process involving three-phase reaction of gas, liquid and solid. To monitor the process and to improve the product quality, as well as to visualize the fault type clearly, a fault diagnosis method based on selforganizing map(SOM) and high dimensional feature extraction method, local tangent space alignment(LTSA),is proposed. In this method, LTSA can reduce the dimension and keep the topology information simultaneously,and SOM distinguishes various states on the output map. Monitoring results of PX oxidation reaction process indicate that the LTSA–SOM can well detect and visualize the fault type.
基金supported by the National Natural Science Foundation of China(No.62072020)Key-Area Research and the Leading Talents in Innovation and Entrepreneurship of Qingdao(No.19-3-2-21-zhc).
文摘Visual localization and object detection both play important roles in various tasks.In many indoor application scenarios where some detected objects have fixed positions,the two techniques work closely together.However,few researchers consider these two tasks simultaneously,because of a lack of datasets and the little attention paid to such environments.In this paper,we explore multi-task network design and joint refinement of detection and localization.To address the dataset problem,we construct a medium indoor scene of an aviation exhibition hall through a semi-automatic process.The dataset provides localization and detection information,and is publicly available at https://drive.google.com/drive/folders/1U28zk0N4_I0db zkqyIAK1A15k9oUKOjI?usp=sharing for benchmarking localization and object detection tasks.Targeting this dataset,we have designed a multi-task network,JLDNet,based on YOLO v3,that outputs a target point cloud and object bounding boxes.For dynamic environments,the detection branch also promotes the perception of dynamics.JLDNet includes image feature learning,point feature learning,feature fusion,detection construction,and point cloud regression.Moreover,object-level bundle adjustment is used to further improve localization and detection accuracy.To test JLDNet and compare it to other methods,we have conducted experiments on 7 static scenes,our constructed dataset,and the dynamic TUM RGB-D and Bonn datasets.Our results show state-of-the-art accuracy for both tasks,and the benefit of jointly working on both tasks is demonstrated.
基金funded by Natural Science Foundation of Hubei Province under Grant Nos.2024AFB273 and 2024AFB139the Hubei Key Laboratory of Power System Design and Test for Electrical Vehicle under Grant No.ZDSYS202425.
文摘To solve the problem of mobile robots needing to adjust their pose for accurate operation after reaching the target point in the indoor environment,a localization method based on scene modeling and recognition has been designed.Firstly,the offline scene model is created by both handcrafted feature and semantic feature.Then,the scene recognition and location calculation are performed online based on the offline scene model.To improve the accuracy of recognition and location calculation,this paper proposes a method that integrates both semantic features matching and handcrafted features matching.Based on the results of scene recognition,the accurate location is obtained through metric calculation with 3D information.The experimental results show that the accuracy of scene recognition is over 90%,and the average localization error is less than 1 meter.Experimental results demonstrate that the localization has a better performance after using the proposed improved method.
基金supported by National Natural Science Foundation of China(Nos.61273352 and 61473295)National High Technology Research and Development Program of China(863 Program)(No.2015AA042307)Beijing Natural Science Foundation(No.4161002)
文摘A new visual measurement method is proposed to estimate three-dimensional （3D） position of the object on the floor based on a single camera. The camera fixed on a robot is in an inclined position with respect to the floor. A measurement model with the camera＇s extrinsic parameters such as the height and pitch angle is described. Single image of a chessboard pattern placed on the floor is enough to calibrate the camera＇s extrinsic parameters after the camera＇s intrinsic parameters are calibrated. Then the position of object on the floor can be computed with the measurement model. Furthermore, the height of object can be calculated with the paired-points in the vertical line sharing the same position on the floor. Compared to the conventional method used to estimate the positions on the plane, this method can obtain the 3D positions. The indoor experiment testifies the accuracy and validity of the proposed method.
基金supported by the National Key Research and Development Program of China under[Grant number 2019YFC1511304]supported by the Pilot Fund of Frontier Science and Disruptive Technology of Aerospace Information Research Institute,Chinese Academy of Sciences under[Grant number E0Z21101].
文摘Simultaneous Localization and Mapping(SLAM)has been widely used in emergency response,self-driving and city-scale 3D mapping and navigation.Recent deep-learning based feature point extractors have demonstrated superior performance in dealing with the complex environmental challenges(e.g.extreme lighting)while the traditional extractors are struggling.In this paper,we have successfully improved the robustness and accuracy of a monocular visual SLAM system under various complex scenes by adding a deep learning based visual localization thread as an augmentation to the visual SLAM framework.In this thread,our feature extractor with an efficient lightweight deep neural network is used for absolute pose and scale estimation in real time using the highly accurate georeferenced prior map database at 20cm geometric accuracy created by our in-house and low-cost LiDAR and camera integrated device.The closed-loop error provided by our SLAM system with and without this enhancement is 1.03m and 18.28m respectively.The scale estimation of the monocular visual SLAM is also significantly improved(0.01 versus 0.98).In addition,a novel camera-LiDAR calibration workflow is also provided for large-scale 3D mapping.This paper demonstrates the application and research potential of deep-learning based vision SLAM with image and LiDAR sensors.
基金Supported by the National Science Foundation of USA (No.IIS-0812027)
文摘Sensitivity analysis is a powerful method for discovering the significant factors that contribute to understanding the interaction between variables in multivariate datasets. A number of sensitivity analysis methods fall into the class of local analysis, in which the sensitivity is defined as the partial derivatives of a target variable with respect to a group of independent variables. In a recent paper, we presented a novel pointwise local pattern exploration system for visual sensitivity analysis. Using this system, analysts are able to explore local patterns and the sensitivity at individual data points, which reveals the relationships between a focal point and its neighbors this paper we present several evaluations of the system, including case studies with real datasets, user studies the effectiveness of the visualizations and interactions, and a detailed description of the experience of a user In on
文摘The extraction and description of image features are very important for visual simultaneous localization and mapping(V-SLAM).A rotated boosted efficient binary local image descriptor(BEBLID)SLAM(RB-SLAM)algorithm based on improved oriented fast and rotated brief(ORB)feature description is proposed in this paper,which can solve the problems of low localization accuracy and time efficiency of the current ORB-SLAM3 algorithm.Firstly,it uses the BEBLID to replace the feature point description algorithm of the original ORB to enhance the expressiveness and description efficiency of the image.Secondly,it adds rotational invariance to the BEBLID using the orientation information of the feature points.It also selects the rotationally stable bits in the BEBLID to further enhance the rotational invariance of the BEBLID.Finally,it retrains the binary visual dictionary based on the BEBLID to reduce the cumulative error of V-SLAM and improve the loading speed of the visual dictionary.Experiments show that the dictionary loading efficiency is improved by more than 10 times.The RB-SLAM algorithm improves the trajectory accuracy by 24.75%on the TUM dataset and 26.25%on the EuRoC dataset compared to the ORB-SLAM3 algorithm.
基金supported by the National Natural Science Foundation of China(Grant No.62220106003)Tsinghua-Tencent Joint Laboratory for Internet Innovation Technology.
文摘The cross-view matching of local image features is a fundamental task in visual localization and 3D reconstruction.This study proposes FilterGNN,a transformer-based graph neural network(GNN),aiming to improve the matching efficiency and accuracy of visual descriptors.Based on high matching sparseness and coarse-to-fine covisible area detection,FilterGNN utilizes cascaded optimal graph-matching filter modules to dynamically reject outlier matches.Moreover,we successfully adapted linear attention in FilterGNN with post-instance normalization support,which significantly reduces the complexity of complete graph learning from O(N2)to O(N).Experiments show that FilterGNN requires only 6%of the time cost and 33.3%of the memory cost compared with SuperGlue under a large-scale input size and achieves a competitive performance in various tasks,such as pose estimation,visual localization,and sparse 3D reconstruction.
