It has been over a decade since the first coded aperture video compressive sensing(CS)system was reported.The underlying principle of this technology is to employ a high-frequency modulator in the optical path to modu...It has been over a decade since the first coded aperture video compressive sensing(CS)system was reported.The underlying principle of this technology is to employ a high-frequency modulator in the optical path to modulate a recorded high-speed scene within one integration time.The superimposed image captured in this manner is modulated and compressed,since multiple modulation patterns are imposed.Following this,reconstruction algorithms are utilized to recover the desired high-speed scene.One leading advantage of video CS is that a single captured measurement can be used to reconstruct a multi-frame video,thereby enabling a low-speed camera to capture high-speed scenes.Inspired by this,a number of variants of video CS systems have been built,mainly using different modulation devices.Meanwhile,in order to obtain high-quality reconstruction videos,many algorithms have been developed,from optimization-based iterative algorithms to deep-learning-based ones.Recently,emerging deep learning methods have been dominant due to their high-speed inference and high-quality reconstruction,highlighting the possibility of deploying video CS in practical applications.Toward this end,this paper reviews the progress that has been achieved in video CS during the past decade.We further analyze the efforts that need to be made—in terms of both hardware and algorithms—to enable real applications.Research gaps are put forward and future directions are summarized to help researchers and engineers working on this topic.展开更多
Recently,deep learning-based video compressive sensing reconstruction(VCSR)technologies have significantly improved reconstructed video quality by taking advantage of spatial and temporal correlations.However,existing...Recently,deep learning-based video compressive sensing reconstruction(VCSR)technologies have significantly improved reconstructed video quality by taking advantage of spatial and temporal correlations.However,existing VcSR work mainly focuses on improving deep learning-based motion compensation without optimizing local and global information,leaving much room for further improvement.This paper proposes a novel VcSR method,JVCSR+,which focuses on optimizing feature information,removing reconstruction artifacts,and increasing the resolution simultaneously.Specifically,the measurement matrix in the proposed compressive sensing(CS)module can learn adaptively,so that sampled measurements retain more image structure information for better reconstruction.An average search module is also proposed to detect more suitable areas for references,thereby attaining superior motion compensation performance.Within the loop,the enhanced frame is utilized as a reference to improve recovery of the current frame.Furthermore,we propose an out-loop super-resolution module for VCSR to obtain high-quality images at low bitrates.The results of extensive experiments demonstrate that our proposed JVcSR+obtains promising performance compared to state-of-the-art CS methods within the same bitrate range.展开更多
Compressed sensing(CS)is a novel technology to acquire and reconstruct sparse signals below the Nyquist rate.It has great potential in image and video acquisition and processing.To effectively improve the sparsity of ...Compressed sensing(CS)is a novel technology to acquire and reconstruct sparse signals below the Nyquist rate.It has great potential in image and video acquisition and processing.To effectively improve the sparsity of signal being measured and reconstructing efficiency,an encoding and decoding model of residual distributed compressive video sensing based on double side information(RDCVS-DSI)is proposed in this paper.Exploiting the characteristics of image itself in the frequency domain and the correlation between successive frames,the model regards the video frame in low quality as the first side information in the process of coding,and generates the second side information for the non-key frames using motion estimation and compensation technology at its decoding end.Performance analysis and simulation experiments show that the RDCVS-DSI model can rebuild the video sequence with high fidelity in the consumption of quite low complexity.About 1~5 dB gain in the average peak signal-to-noise ratio of the reconstructed frames is observed,and the speed is close to the least complex DCVS,when compared with prior works on compressive video sensing.展开更多
In this paper,a video compressed sensing reconstruction algorithm based on multidimensional reference frames is proposed using the sparse characteristics of video signals in different sparse representation domains.Fir...In this paper,a video compressed sensing reconstruction algorithm based on multidimensional reference frames is proposed using the sparse characteristics of video signals in different sparse representation domains.First,the overall structure of the proposed video compressed sensing algorithm is introduced in this paper.The paper adopts a multi-reference frame bidirectional prediction hypothesis optimization algorithm.Then,the paper proposes a reconstruction method for CS frames at the re-decoding end.In addition to using key frames of each GOP reconstructed in the time domain as reference frames for reconstructing CS frames,half-pixel reference frames and scaled reference frames in the pixel domain are also used as CS frames.Reference frames of CS frames are used to obtain higher quality assumptions.Themethod of obtaining reference frames in the pixel domain is also discussed in detail in this paper.Finally,the reconstruction algorithm proposed in this paper is compared with video compression algorithms in the literature that have better reconstruction results.Experiments show that the algorithm has better performance than the best multi-reference frame video compression sensing algorithm and can effectively improve the quality of slowmotion video reconstruction.展开更多
基金supported by the National Natural Science Foundation of China(61931012,62171258,62088102,and 62271414)the Zhejiang Provincial Outstanding Youth Science Foundation(LR23F010001)the Key Project of Westlake Institute for Optoelectronics(2023GD007).
文摘It has been over a decade since the first coded aperture video compressive sensing(CS)system was reported.The underlying principle of this technology is to employ a high-frequency modulator in the optical path to modulate a recorded high-speed scene within one integration time.The superimposed image captured in this manner is modulated and compressed,since multiple modulation patterns are imposed.Following this,reconstruction algorithms are utilized to recover the desired high-speed scene.One leading advantage of video CS is that a single captured measurement can be used to reconstruct a multi-frame video,thereby enabling a low-speed camera to capture high-speed scenes.Inspired by this,a number of variants of video CS systems have been built,mainly using different modulation devices.Meanwhile,in order to obtain high-quality reconstruction videos,many algorithms have been developed,from optimization-based iterative algorithms to deep-learning-based ones.Recently,emerging deep learning methods have been dominant due to their high-speed inference and high-quality reconstruction,highlighting the possibility of deploying video CS in practical applications.Toward this end,this paper reviews the progress that has been achieved in video CS during the past decade.We further analyze the efforts that need to be made—in terms of both hardware and algorithms—to enable real applications.Research gaps are put forward and future directions are summarized to help researchers and engineers working on this topic.
基金supported by JSPS KAKENHI Grant Number JP22K12101。
文摘Recently,deep learning-based video compressive sensing reconstruction(VCSR)technologies have significantly improved reconstructed video quality by taking advantage of spatial and temporal correlations.However,existing VcSR work mainly focuses on improving deep learning-based motion compensation without optimizing local and global information,leaving much room for further improvement.This paper proposes a novel VcSR method,JVCSR+,which focuses on optimizing feature information,removing reconstruction artifacts,and increasing the resolution simultaneously.Specifically,the measurement matrix in the proposed compressive sensing(CS)module can learn adaptively,so that sampled measurements retain more image structure information for better reconstruction.An average search module is also proposed to detect more suitable areas for references,thereby attaining superior motion compensation performance.Within the loop,the enhanced frame is utilized as a reference to improve recovery of the current frame.Furthermore,we propose an out-loop super-resolution module for VCSR to obtain high-quality images at low bitrates.The results of extensive experiments demonstrate that our proposed JVcSR+obtains promising performance compared to state-of-the-art CS methods within the same bitrate range.
基金Supported by National Natural Science Foundation of China(61170147)Major Cooperation Project of Production and College in Fujian Province(2012H61010016)Natural Science Foundation of Fujian Province(2013J01234)
文摘Compressed sensing(CS)is a novel technology to acquire and reconstruct sparse signals below the Nyquist rate.It has great potential in image and video acquisition and processing.To effectively improve the sparsity of signal being measured and reconstructing efficiency,an encoding and decoding model of residual distributed compressive video sensing based on double side information(RDCVS-DSI)is proposed in this paper.Exploiting the characteristics of image itself in the frequency domain and the correlation between successive frames,the model regards the video frame in low quality as the first side information in the process of coding,and generates the second side information for the non-key frames using motion estimation and compensation technology at its decoding end.Performance analysis and simulation experiments show that the RDCVS-DSI model can rebuild the video sequence with high fidelity in the consumption of quite low complexity.About 1~5 dB gain in the average peak signal-to-noise ratio of the reconstructed frames is observed,and the speed is close to the least complex DCVS,when compared with prior works on compressive video sensing.
文摘In this paper,a video compressed sensing reconstruction algorithm based on multidimensional reference frames is proposed using the sparse characteristics of video signals in different sparse representation domains.First,the overall structure of the proposed video compressed sensing algorithm is introduced in this paper.The paper adopts a multi-reference frame bidirectional prediction hypothesis optimization algorithm.Then,the paper proposes a reconstruction method for CS frames at the re-decoding end.In addition to using key frames of each GOP reconstructed in the time domain as reference frames for reconstructing CS frames,half-pixel reference frames and scaled reference frames in the pixel domain are also used as CS frames.Reference frames of CS frames are used to obtain higher quality assumptions.Themethod of obtaining reference frames in the pixel domain is also discussed in detail in this paper.Finally,the reconstruction algorithm proposed in this paper is compared with video compression algorithms in the literature that have better reconstruction results.Experiments show that the algorithm has better performance than the best multi-reference frame video compression sensing algorithm and can effectively improve the quality of slowmotion video reconstruction.
