Background Free-viewpoint video(FVV)is processed video content in which viewers can freely select the viewing position and angle.FVV delivers an improved visual experience and can also help synthesize special effects ...Background Free-viewpoint video(FVV)is processed video content in which viewers can freely select the viewing position and angle.FVV delivers an improved visual experience and can also help synthesize special effects and virtual reality content.In this paper,a complete FVV system is proposed to interactively control the viewpoints of video relay programs through multimedia terminals such as computers and tablets.Methods The hardware of the FVV generation system is a set of synchronously controlled cameras,and the software generates videos in novel viewpoints from the captured video using view interpolation.The interactive interface is designed to visualize the generated video in novel viewpoints and enable the viewpoint to be changed interactively.Results Experiments show that our system can synthesize plausible videos in intermediate viewpoints with a view range of up to 180°.展开更多
In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. ...In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. We present a new VBR system that creates new views of a live dynamic scene. This system provides high quality images and does not require any background subtraction. Our method follows a plane-sweep approach and reaches real-time rendering using consumer graphic hardware, graphics processing unit （GPU）. Only one computer is used for both acquisition and rendering. The video stream acquisition is performed by at least 3 webcams. We propose an additional video stream management that extends the number of webcams to 10 or more. These considerations make our system low-cost and hence accessible for everyone. We also present an adaptation of our plane-sweep method to create simultaneously multiple views of the scene in real-time. Our system is especially designed for stereovision using autostereoscopic displays. The new views are computed from 4 webcams connected to a computer and are compressed in order to be transfered to a mobile phone. Using GPU programming, our method provides up to 16 images of the scene in real-time. The use of both GPU and CPU makes this method work on only one consumer grade computer.展开更多
A semi-reference image quality assessment metric based on similarity measurement for synthesized virtual viewpoint image （VVI） in free-viewpoint television system （FFV） is proposed in this paper. The key point of ...A semi-reference image quality assessment metric based on similarity measurement for synthesized virtual viewpoint image （VVI） in free-viewpoint television system （FFV） is proposed in this paper. The key point of the proposed metric is taking resemblant information between VVI and its neighbor view images for quality assessment to make our metric to be extended to multi-semi-reference image quality assessment easily. The proposed metric first extracts impact factors from image features, then combines an image synthesis technique and similarity functions, in which, disparity information are taken into account for registering the resemblant regions. Experiments are divided into three phases. Phase I is to verify the validation of the proposed metric by taking impaired images and original reference into account. The experimental results show the agreement between evaluation scores and bio-characteristic of human visual system. Phase II shows the accordance with Phase I by taking neighbor view as reference. The proposed metric can be taken as a full reference one to evaluate the image quality even though the original reference is absent. Phase III is then performed to evaluate the quality of WI. Evaluation scores in the experimental results are able to evaluate the quality of VVI.展开更多
Free-viewpoint video allows the user to view objects from any virtual perspective,creating an immersive visual experience.This technology enhances the interactivity and freedom of multimedia performances.However,many ...Free-viewpoint video allows the user to view objects from any virtual perspective,creating an immersive visual experience.This technology enhances the interactivity and freedom of multimedia performances.However,many free-viewpoint video synthesis methods hardly satisfy the requirement to work in real time with high precision,particularly for sports fields having large areas and numerous moving objects.To address these issues,we propose a freeviewpoint video synthesis method based on distance field acceleration.The central idea is to fuse multiview distance field information and use it to adjust the search step size adaptively.Adaptive step size search is used in two ways:for fast estimation of multiobject three-dimensional surfaces,and synthetic view rendering based on global occlusion judgement.We have implemented our ideas using parallel computing for interactive display,using CUDA and OpenGL frameworks,and have used real-world and simulated experimental datasets for evaluation.The results show that the proposed method can render free-viewpoint videos with multiple objects on large sports fields at 25 fps.Furthermore,the visual quality of our synthetic novel viewpoint images exceeds that of state-of-the-art neural-rendering-based methods.展开更多
目的基于深度图的绘制(DIBR)是一种新型的虚拟视点生成技术,在诸多方面得到了广泛的应用。然而,该技术还不能满足实时性的绘制需求。为了在保证绘制质量不下降的前提下,尽可能地提高绘制速度,提出了一种高效的3D-Warping(3维坐标变换)...目的基于深度图的绘制(DIBR)是一种新型的虚拟视点生成技术,在诸多方面得到了广泛的应用。然而,该技术还不能满足实时性的绘制需求。为了在保证绘制质量不下降的前提下,尽可能地提高绘制速度,提出了一种高效的3D-Warping(3维坐标变换)算法。方法主要在以下3个方面进行了改进:1)引入了深度—视差映射表技术,避免了重复地进行视差求取操作。2)对深度平坦的像素块进行基于块的3D-Warping,减少了映射的次数。对深度非平坦像素块中的像素点采取传统的基于像素点的3D-Warping,保证了映射的准确性。3)针对两种不同的3D-Warping方式,分别提出了相应的插值算法。在水平方向上,改进的像素插值算法对紧邻插值和Splatting(散射)插值算法进行了折中,只在映射像素点与待插值像素点很近的情况下才进行紧邻插值,否则进行Splatting插值;在深度方向上,它对Z-Buffer(深度缓存)技术进行了改进,舍弃了与前景物体太远的映射像素点,而对其他映射像素点按深度值进行加权操作。结果实验结果表明,与标准绘制方案的整像素精度相比,绘制时间平均节省了72.05%;与标准绘制方案的半像素精度相比,PSNR平均提高了0.355 d B,SSIM平均提高了0.001 15。结论改进算法非常适用于水平设置相机系统的DIBR技术中的整像素精度绘制,对包含大量深度平坦区域的视频序列效果明显,不但能够提高绘制的速度,而且可以有效地改善绘制的客观质量。展开更多
The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or obje...The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or object has been added.In particular,the free-view service is an extended concept of multi-view and provides a freer viewpoint.However,since numerous videos and additional data are required for its construction,all of the clips constituting the content cannot be simultaneously provided.Only certain clips are selected and provided to the user.If the video is not the preferred video,change request is made,and a delay occurs during retransmission from the server.Delays due to frequent re-requests degrade the overall quality of service.For free-view services,selectively transmitting the video according to the user’s desired viewpoint and region of interest within the limited network of available videos is important.In this study,we propose a method of screening and providing the correct video based on objects in the contents.Based on the method of recognizing the object in each clip,we designed a method of setting its priority based on information about the object’s location for each viewpoint.During the transmission and receiving process using this information,the selected video can be rapidly recognized and changed.Herein,we present a service system configuration method and propose video selection examples for free-view services.展开更多
基金Supported by USDA 2018-67021-27416NSFJS BK20190292.
文摘Background Free-viewpoint video(FVV)is processed video content in which viewers can freely select the viewing position and angle.FVV delivers an improved visual experience and can also help synthesize special effects and virtual reality content.In this paper,a complete FVV system is proposed to interactively control the viewpoints of video relay programs through multimedia terminals such as computers and tablets.Methods The hardware of the FVV generation system is a set of synchronously controlled cameras,and the software generates videos in novel viewpoints from the captured video using view interpolation.The interactive interface is designed to visualize the generated video in novel viewpoints and enable the viewpoint to be changed interactively.Results Experiments show that our system can synthesize plausible videos in intermediate viewpoints with a view range of up to 180°.
基金This work was supported by Foundation of Technology Supporting the Creation of Digital Media Contents project (CREST, JST), Japan
文摘In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. We present a new VBR system that creates new views of a live dynamic scene. This system provides high quality images and does not require any background subtraction. Our method follows a plane-sweep approach and reaches real-time rendering using consumer graphic hardware, graphics processing unit （GPU）. Only one computer is used for both acquisition and rendering. The video stream acquisition is performed by at least 3 webcams. We propose an additional video stream management that extends the number of webcams to 10 or more. These considerations make our system low-cost and hence accessible for everyone. We also present an adaptation of our plane-sweep method to create simultaneously multiple views of the scene in real-time. Our system is especially designed for stereovision using autostereoscopic displays. The new views are computed from 4 webcams connected to a computer and are compressed in order to be transfered to a mobile phone. Using GPU programming, our method provides up to 16 images of the scene in real-time. The use of both GPU and CPU makes this method work on only one consumer grade computer.
基金Supported by the National Natural Science Foundation of China (No. 60672073,60872094)the Program for New Century Excellent Talents in University (NCET-06-0537)the Natural Science Foundation of Ningbo (No. 2007A610037).
文摘A semi-reference image quality assessment metric based on similarity measurement for synthesized virtual viewpoint image （VVI） in free-viewpoint television system （FFV） is proposed in this paper. The key point of the proposed metric is taking resemblant information between VVI and its neighbor view images for quality assessment to make our metric to be extended to multi-semi-reference image quality assessment easily. The proposed metric first extracts impact factors from image features, then combines an image synthesis technique and similarity functions, in which, disparity information are taken into account for registering the resemblant regions. Experiments are divided into three phases. Phase I is to verify the validation of the proposed metric by taking impaired images and original reference into account. The experimental results show the agreement between evaluation scores and bio-characteristic of human visual system. Phase II shows the accordance with Phase I by taking neighbor view as reference. The proposed metric can be taken as a full reference one to evaluate the image quality even though the original reference is absent. Phase III is then performed to evaluate the quality of WI. Evaluation scores in the experimental results are able to evaluate the quality of VVI.
基金supported by the National Natural Science Foundation of China(Nos.62172315,62073262,and 61672429)the Fundamental Research Funds for the Central Universities,the Innovation Fund of Xidian University(No.20109205456)the Key Research and Development Program of Shaanxi(No.S2021-YF-ZDCXL-ZDLGY-0127),and HUAWEI.
文摘Free-viewpoint video allows the user to view objects from any virtual perspective,creating an immersive visual experience.This technology enhances the interactivity and freedom of multimedia performances.However,many free-viewpoint video synthesis methods hardly satisfy the requirement to work in real time with high precision,particularly for sports fields having large areas and numerous moving objects.To address these issues,we propose a freeviewpoint video synthesis method based on distance field acceleration.The central idea is to fuse multiview distance field information and use it to adjust the search step size adaptively.Adaptive step size search is used in two ways:for fast estimation of multiobject three-dimensional surfaces,and synthetic view rendering based on global occlusion judgement.We have implemented our ideas using parallel computing for interactive display,using CUDA and OpenGL frameworks,and have used real-world and simulated experimental datasets for evaluation.The results show that the proposed method can render free-viewpoint videos with multiple objects on large sports fields at 25 fps.Furthermore,the visual quality of our synthetic novel viewpoint images exceeds that of state-of-the-art neural-rendering-based methods.
文摘目的基于深度图的绘制(DIBR)是一种新型的虚拟视点生成技术,在诸多方面得到了广泛的应用。然而,该技术还不能满足实时性的绘制需求。为了在保证绘制质量不下降的前提下,尽可能地提高绘制速度,提出了一种高效的3D-Warping(3维坐标变换)算法。方法主要在以下3个方面进行了改进:1)引入了深度—视差映射表技术,避免了重复地进行视差求取操作。2)对深度平坦的像素块进行基于块的3D-Warping,减少了映射的次数。对深度非平坦像素块中的像素点采取传统的基于像素点的3D-Warping,保证了映射的准确性。3)针对两种不同的3D-Warping方式,分别提出了相应的插值算法。在水平方向上,改进的像素插值算法对紧邻插值和Splatting(散射)插值算法进行了折中,只在映射像素点与待插值像素点很近的情况下才进行紧邻插值,否则进行Splatting插值;在深度方向上,它对Z-Buffer(深度缓存)技术进行了改进,舍弃了与前景物体太远的映射像素点,而对其他映射像素点按深度值进行加权操作。结果实验结果表明,与标准绘制方案的整像素精度相比,绘制时间平均节省了72.05%;与标准绘制方案的半像素精度相比,PSNR平均提高了0.355 d B,SSIM平均提高了0.001 15。结论改进算法非常适用于水平设置相机系统的DIBR技术中的整像素精度绘制,对包含大量深度平坦区域的视频序列效果明显,不但能够提高绘制的速度,而且可以有效地改善绘制的客观质量。
基金supported by the Basic Science Research Program through the National Research Foundation of Korea(NRF)funded by the Ministry of Education(NRF-2019R1F1A1061635)by a research grant from Seoul Women’s University(2020-0213).
文摘The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or object has been added.In particular,the free-view service is an extended concept of multi-view and provides a freer viewpoint.However,since numerous videos and additional data are required for its construction,all of the clips constituting the content cannot be simultaneously provided.Only certain clips are selected and provided to the user.If the video is not the preferred video,change request is made,and a delay occurs during retransmission from the server.Delays due to frequent re-requests degrade the overall quality of service.For free-view services,selectively transmitting the video according to the user’s desired viewpoint and region of interest within the limited network of available videos is important.In this study,we propose a method of screening and providing the correct video based on objects in the contents.Based on the method of recognizing the object in each clip,we designed a method of setting its priority based on information about the object’s location for each viewpoint.During the transmission and receiving process using this information,the selected video can be rapidly recognized and changed.Herein,we present a service system configuration method and propose video selection examples for free-view services.
