摘要
视频超分方法对多幅低分辨图像进行融合、运动补偿等处理,重构一幅高分辨率图像。基于时空注意力的双分支视频超分辨率网络对运动补偿后的低分辨率图像分配不同的权重,并且引入梯度信息避免超分辨率图像出现结构失真等问题。本网络由一个重建高分辨率图像的图像分支和一个产生高分辨率梯度图的梯度分支组成。图像分支采用一组运动补偿后的低分辨率图像作为输入,并使用时空注意力残差块突出显示信息丰富的局部区域以及关键的时间通道,来恢复丢失的高频细节。梯度分支根据运动补偿后低分辨率图像的梯度图学习显著的结构信息。这两个分支通过融合不同层次的中间特征相互提供先验知识。最后将梯度分支重构的高频结构信息反馈到图像分支,进一步提高超分辨率视频帧的结构一致性。在Vid4和SPMCS-11数据集上验证了本模型的有效性。
The goal of video super-resolution is to recover a high-resolution video frame from multiple low-resolution frames by using fusion,motion compensation,and other techniques.The spatial-temporal attention-based two-branch video super-resolution network assigns different weights to motion-compensated low-resolution frames,and introduces gradient information to avoid the problem of structural distortion.The proposed network consisted of an image branch that reconstructed a high-resolution video frame and a gradient branch that generated a high-resolution gradient map.The image branch took a set of motion-compensated low-resolution frames as input,and employed STAR(Spatial-Temporal Attention Residual)blocks to highlight informative local regions as well as critical temporal channels for recovering the lost high-frequency details.The gradient branch learned the salient structural information according to the gradient maps of motion-compensated low-resolution frames.The two branches provided prior knowledge to each other by fusing intermediate features at different levels.At last,the high-frequency structural details reconstructed by the gradient branch were fed back to the image branch to further promote the structural consistency of HR frames.Experimental results on the Vid4 and the SPMCS-11 datasets demonstrated the effectiveness of the proposed method.
作者
刘翠微
丁彩胤
李侃
颜卓
石祥滨
LIU Cui-wei;DING Cai-yin;LI Kan;YAN Zhuo;SHI Xiang-bin(College of Computer Science,Shenyang Aerospace University,Shenyang 110136,China)
出处
《沈阳航空航天大学学报》
2022年第1期45-53,共9页
Journal of Shenyang Aerospace University
基金
国家自然科学基金(项目编号:61602320)辽宁省自然基金(项目编号:2021-MS-266)
辽宁省教育厅科学研究经费面上项目(项目编号:LJKZ0220)
沈阳市中青年科技创新人才支持计划(项目编号:RC210427)
沈阳航空航天大学博士启动基金(项目编号:18YB39)。
关键词
视频超分辨
双分支视频超分辨率网络
时空注意力
梯度图
结构一致性
video super-resolution
two-branch video super-resolution network
spatial-temporal attention
gradient map
structural consistency
