摘要
深度图像能够有效描述三维场景的信息,然而由于采集设备的局限性和不理想的成像环境,深度传感器获取的深度图像往往分辨率较低、高频信息较少,提高深度图像的分辨率具有重要意义。部分深度图超分辨率算法通过引入同一场景下的RGB图像为深度图超分辨率过程提供指导信息,显著提升了算法性能。如何充分、有效地利用RGB信息,改善深度图和RGB图像的模态不一致性,引导深度图超分辨率重建过程极具挑战。已有方法多关注于高频信息,忽略了低频全局的信息,影响了算法性能的提升。对此,提出了彩色图像引导的、高低频特征调制融合的深度图像超分辨率重建算法。具体地,设计了一个双分支特征提取模块,分别针对彩色图像和深度图像进行高低频特征提取,在各个分支采用CNN和Transformer分别提取局部高频和全局低频信息,通过构造双向调制模块,实现对彩色和深度图像高频信息之间和低频信息之间的双向转换与融合。模型经过不同模态不同频率内的双向调制及后续高低频信息的融合,充分挖掘深度图像与彩色图像之间的互补信息,使得基于彩色图像引导的深度超分辨率算法能够取得更好的重建效果。另外,利用可逆神经网络INN进行无损信息压缩,以更好地提取高频细节信息,采用四叉树注意力机制有效降低了Transformer提取全局信息的计算复杂度,提高了算法效率。在公开数据集上进行了实验,结果表明,所提方法在定量和定性两方面均优于对比方法,取得了较好的主观视觉效果。
Depth images effectively describe the information of a 3D scene.However,the acquisition equipment and imaging environment limit the resolution and high-frequency information of the depth images acquired by depth sensors.It is imperative to improve the resolution of depth images.Some depth map super-resolution algorithms have significantly improved their performance by introducing RGB images from the same scene to provide guidance information for the depth map super-resolution process.The key challenge lies in effectively leveraging the RGB information to guide the depth map super-resolution reconstruction process,addressing the modal inconsistency between the depth map and RGB images.Existing methods primarily focus on high-frequency information,overlooking the low-frequency global information crucial for algorithm performance.To address these limitations,this paper proposes a novel color image-guided,high and low-frequency feature modulation fusion super-resolution reconstruction algorithm for depth maps.A two-branch feature extraction module extracts high and low frequency features from color and depth images,respectively.CNN and Transformer are used in each branch to extract local high frequency and global low frequency information.A two-way transformation and fusion between high frequency information and low frequency information of color and depth images is achieved by constructing a two-way modulation module.The model fully exploits the complementary information between the depth image and the color image.It uses a bidirectional modulation within different modes and different frequencies and the subsequent fusion of high and low-frequency information.The depth super-resolution algorithm based on the guidance of the color image can achieve better reconstruction results.The lossless information compression using reversible neural network INN extracts high-frequency detail information more effectively,and the quadtree attention mechanism reduces the computational complexity of the Transformer in extracting global information,improving the efficiency of the algorithm.The experimental results on the public datasets show that the proposed method outperforms the comparison methods in both quantitative and qualitative aspects,achieving better subjective visualization results.
作者
徐晗智
李嘉莹
梁宇栋
魏巍
XU Hanzhi;LI Jiaying;LIANG Yudong;WEI Wei(School of Mathematical Science,Shanxi University,Taiyuan 030006,China;School of Computer and Information Technology,Shanxi University,Taiyuan 030006,China;Key Laboratory of Ministry of Education for Computation Intelligence and Chinese Information Processing,Shanxi University,Taiyuan 030006,China)
出处
《计算机科学》
北大核心
2025年第6期228-238,共11页
Computer Science
基金
国家自然科学基金(61802237,62272284)
山西省基础研究计划(自由探索类)项目(202203021221002,202203021211291)
山西省自然科学基金(201901D211176,202103021223464)
山西省高等学校科技创新项目(2019L0066)
山西省科技重大专项计划(202101020101019)
山西省重点研发计划(202102070301019)
山西省科技创新青年人才团队项目(202204051001015)。
关键词
深度图超分重建
混合特征
双向调制
四叉树注意力机制
Depth image super-resolution reconstruction
Hybrid features
Bidirectional modulation
Quadtree attention mechanics
