Existing depth completion methods are often targeted at a specific sparse depth type and generalize poorly across task domains.We present a method to complete sparse/semi-dense,noisy,and potentially low-resolution dep...Existing depth completion methods are often targeted at a specific sparse depth type and generalize poorly across task domains.We present a method to complete sparse/semi-dense,noisy,and potentially low-resolution depth maps obtained by various range sensors,including those in modern mobile phones,or by multi-view reconstruction algorithms.Our method leverages a data-driven prior in the form of a single image depth prediction network trained on large-scale datasets,the output of which is used as an input to our model.We propose an effective training scheme where we simulate various sparsity patterns in typical task domains.In addition,we design two new benchmarks to evaluate the generalizability and robustness of depth completion methods.Our simple method shows superior cross-domain generalization ability against state-of-the-art depth completion methods,introducing a practical solution to highqualitydepthcapture onamobile device.展开更多
Highly scattering media,such as milk,skin,and clouds,are common in the real world.Rendering participating media is challenging,especially for highorder scattering dominant media,because the light may undergo a large n...Highly scattering media,such as milk,skin,and clouds,are common in the real world.Rendering participating media is challenging,especially for highorder scattering dominant media,because the light may undergo a large number of scattering events before leaving the surface.Monte Carlo-based methods typically require a long time to produce noise-free results.Based on the observation that low-albedo media contain less noise than high-albedo media,we propose reducing the variance of the rendered results using differentiable regularization.We first render an image with low-albedo participating media together with the gradient with respect to the albedo,and then predict the final rendered image with a low-albedo image and gradient image via a novel prediction function.To achieve high quality,we also consider the gradients of neighboring frames to provide a noise-free gradient image.Ultimately,our method can produce results with much less overall eror than equal-time path tracing methods.展开更多
For many social events such as public performances, multiple hand-held cameras may capture the same event. This footage is often collected by amateur cinematographers who typically have little control over the scene a...For many social events such as public performances, multiple hand-held cameras may capture the same event. This footage is often collected by amateur cinematographers who typically have little control over the scene and may not pay close attention to the camera. For these reasons, each individually captured video may fail to cover the whole time of the event, or may lose track of interesting foreground content such as a performer. We introduce a new algorithm that can synthesize a single smooth video sequence of moving foreground objects captured by multiple hand-held cameras. This allows later viewers to gain a cohesive narrative experience that can transition between different cameras, even though the input footage may be less than ideal. We first introduce a graph-based method for selecting a good transition route. This allows us to automatically select good cut points for the hand-held videos, so that smooth transitions can be created between the resulting video shots. We also propose a method to synthesize a smooth photorealistic transition video between each pair of hand-held cameras, which preserves dynamic foreground content during this transition. Our experiments demonstrate that our method outperforms previous state-of-the-art methods, which struggle to preserve dynamic foreground content.展开更多
Realistic human skin rendering has been a long-standing challenge in computer graphics.Recently,biophysical-based skin rendering has received increasing attention,as it provides a more realistic skin-rendering and a m...Realistic human skin rendering has been a long-standing challenge in computer graphics.Recently,biophysical-based skin rendering has received increasing attention,as it provides a more realistic skin-rendering and a more intuitive way to adjust the skin style.In this work,we present a novel heterogeneous biophysical-based volume rendering method for human skin that improves the realism of skin appearance while easily simulating various types of skin effects,including skin diseases,by modifying biological coefficient textures.Specifically,we introduce a two-layer skin representation by mesh deformation that explicitly models the epidermis and dermis with heterogeneous volumetric medium layers containing the corresponding spatially varying melanin and hemoglobin,respectively.Furthermore,to better facilitate skin acquisition,we introduced a learning-based framework that automatically estimates spatially varying biological coefficients from an albedo texture,enabling biophysical-based and intuitive editing,such as tanning,pathological vitiligo,and freckles.We illustrated the effects of multiple skinediting applications and demonstrated superior quality to the commonly used random walk skin-rendering method,with more convincing skin details regarding subsurface scattering.展开更多
文摘Existing depth completion methods are often targeted at a specific sparse depth type and generalize poorly across task domains.We present a method to complete sparse/semi-dense,noisy,and potentially low-resolution depth maps obtained by various range sensors,including those in modern mobile phones,or by multi-view reconstruction algorithms.Our method leverages a data-driven prior in the form of a single image depth prediction network trained on large-scale datasets,the output of which is used as an input to our model.We propose an effective training scheme where we simulate various sparsity patterns in typical task domains.In addition,we design two new benchmarks to evaluate the generalizability and robustness of depth completion methods.Our simple method shows superior cross-domain generalization ability against state-of-the-art depth completion methods,introducing a practical solution to highqualitydepthcapture onamobile device.
基金supported by the National Natural Science Foundation of China under Grant No.62172220。
文摘Highly scattering media,such as milk,skin,and clouds,are common in the real world.Rendering participating media is challenging,especially for highorder scattering dominant media,because the light may undergo a large number of scattering events before leaving the surface.Monte Carlo-based methods typically require a long time to produce noise-free results.Based on the observation that low-albedo media contain less noise than high-albedo media,we propose reducing the variance of the rendered results using differentiable regularization.We first render an image with low-albedo participating media together with the gradient with respect to the albedo,and then predict the final rendered image with a low-albedo image and gradient image via a novel prediction function.To achieve high quality,we also consider the gradients of neighboring frames to provide a noise-free gradient image.Ultimately,our method can produce results with much less overall eror than equal-time path tracing methods.
基金supported by a Research Establishment Grant of Victoria University of Wellington (Project No. 8-1620-216786-3744)a Victoria Research Excellence Award
文摘For many social events such as public performances, multiple hand-held cameras may capture the same event. This footage is often collected by amateur cinematographers who typically have little control over the scene and may not pay close attention to the camera. For these reasons, each individually captured video may fail to cover the whole time of the event, or may lose track of interesting foreground content such as a performer. We introduce a new algorithm that can synthesize a single smooth video sequence of moving foreground objects captured by multiple hand-held cameras. This allows later viewers to gain a cohesive narrative experience that can transition between different cameras, even though the input footage may be less than ideal. We first introduce a graph-based method for selecting a good transition route. This allows us to automatically select good cut points for the hand-held videos, so that smooth transitions can be created between the resulting video shots. We also propose a method to synthesize a smooth photorealistic transition video between each pair of hand-held cameras, which preserves dynamic foreground content during this transition. Our experiments demonstrate that our method outperforms previous state-of-the-art methods, which struggle to preserve dynamic foreground content.
基金supported by Key R&D Program of Zhejiang Province(No.2023C01039).
文摘Realistic human skin rendering has been a long-standing challenge in computer graphics.Recently,biophysical-based skin rendering has received increasing attention,as it provides a more realistic skin-rendering and a more intuitive way to adjust the skin style.In this work,we present a novel heterogeneous biophysical-based volume rendering method for human skin that improves the realism of skin appearance while easily simulating various types of skin effects,including skin diseases,by modifying biological coefficient textures.Specifically,we introduce a two-layer skin representation by mesh deformation that explicitly models the epidermis and dermis with heterogeneous volumetric medium layers containing the corresponding spatially varying melanin and hemoglobin,respectively.Furthermore,to better facilitate skin acquisition,we introduced a learning-based framework that automatically estimates spatially varying biological coefficients from an albedo texture,enabling biophysical-based and intuitive editing,such as tanning,pathological vitiligo,and freckles.We illustrated the effects of multiple skinediting applications and demonstrated superior quality to the commonly used random walk skin-rendering method,with more convincing skin details regarding subsurface scattering.
