3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene,including the scene layout,objects bounding boxes and shapes.Existing holistic sce...3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene,including the scene layout,objects bounding boxes and shapes.Existing holistic scene understanding methods primarily recover scenes from single images,with a focus on indoor scenes.Due to the complexity of real-world,the information provided by a single image is limited,resulting in issues such as object occlusion and omission.Furthermore,captured data from outdoor scenes exhibits characteristics of sparsity,strong temporal dependencies and a lack of annotations.Consequently,the task of understanding and reconstructing outdoor scenes is highly challenging.The authors propose a sparse multi-view images-based 3D scene reconstruction framework(SMSR).It divides the scene reconstruction task into three stages:initial prediction,refinement,and fusion stage.The first two stages extract 3D scene representations from each viewpoint,while the final stage involves selection,calibration and fusion of object positions and orientations across different viewpoints.SMSR effectively address the issue of object omission by utilizing small-scale sequential scene information.Experimental results on the general outdoor scene dataset UrbanScene3D-Art Sci and our proprietary dataset Software College Aerial Time-series Images,demonstrate that SMSR achieves superior performance in the scene understanding and reconstruction.展开更多
Reconstructing dynamic scenes with commodity depth cameras has many applications in computer graphics,computer vision,and robotics.However,due to the presence of noise and erroneous observations from data capturing de...Reconstructing dynamic scenes with commodity depth cameras has many applications in computer graphics,computer vision,and robotics.However,due to the presence of noise and erroneous observations from data capturing devices and the inherently ill-posed nature of non-rigid registration with insufficient information,traditional approaches often produce low-quality geometry with holes,bumps,and misalignments.We propose a novel 3D dynamic reconstruction system,named HDR-Net-Fusion,which learns to simultaneously reconstruct and refine the geometry on the fly with a sparse embedded deformation graph of surfels,using a hierarchical deep reinforcement(HDR)network.The latter comprises two parts:a global HDR-Net which rapidly detects local regions with large geometric errors,and a local HDR-Net serving as a local patch refinement operator to promptly complete and enhance such regions.Training the global HDR-Net is formulated as a novel reinforcement learning problem to implicitly learn the region selection strategy with the goal of improving the overall reconstruction quality.The applicability and efficiency of our approach are demonstrated using a large-scale dynamic reconstruction dataset.Our method can reconstruct geometry with higher quality than traditional methods.展开更多
基金National Key R&D Program of China,Grant/Award Number:2021YFC3300203TaiShan Scholars Program,Grant/Award Number:tsqn202211289+1 种基金Oversea Innovation Team Project of the“20 Regulations for New Universities”funding program of Jinan,Grant/Award Number:2021GXRC073Excellent Youth Scholars Program of Shandong Province,Grant/Award Number:2022HWYQ-048。
文摘3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene,including the scene layout,objects bounding boxes and shapes.Existing holistic scene understanding methods primarily recover scenes from single images,with a focus on indoor scenes.Due to the complexity of real-world,the information provided by a single image is limited,resulting in issues such as object occlusion and omission.Furthermore,captured data from outdoor scenes exhibits characteristics of sparsity,strong temporal dependencies and a lack of annotations.Consequently,the task of understanding and reconstructing outdoor scenes is highly challenging.The authors propose a sparse multi-view images-based 3D scene reconstruction framework(SMSR).It divides the scene reconstruction task into three stages:initial prediction,refinement,and fusion stage.The first two stages extract 3D scene representations from each viewpoint,while the final stage involves selection,calibration and fusion of object positions and orientations across different viewpoints.SMSR effectively address the issue of object omission by utilizing small-scale sequential scene information.Experimental results on the general outdoor scene dataset UrbanScene3D-Art Sci and our proprietary dataset Software College Aerial Time-series Images,demonstrate that SMSR achieves superior performance in the scene understanding and reconstruction.
基金This work was supported by the National Natural Science Foundation of China(Grant Nos.61902210 and 61521002).
文摘Reconstructing dynamic scenes with commodity depth cameras has many applications in computer graphics,computer vision,and robotics.However,due to the presence of noise and erroneous observations from data capturing devices and the inherently ill-posed nature of non-rigid registration with insufficient information,traditional approaches often produce low-quality geometry with holes,bumps,and misalignments.We propose a novel 3D dynamic reconstruction system,named HDR-Net-Fusion,which learns to simultaneously reconstruct and refine the geometry on the fly with a sparse embedded deformation graph of surfels,using a hierarchical deep reinforcement(HDR)network.The latter comprises two parts:a global HDR-Net which rapidly detects local regions with large geometric errors,and a local HDR-Net serving as a local patch refinement operator to promptly complete and enhance such regions.Training the global HDR-Net is formulated as a novel reinforcement learning problem to implicitly learn the region selection strategy with the goal of improving the overall reconstruction quality.The applicability and efficiency of our approach are demonstrated using a large-scale dynamic reconstruction dataset.Our method can reconstruct geometry with higher quality than traditional methods.
