With the popularity of the digital human body,monocular three-dimensional(3D)face reconstruction is widely used in fields such as animation and face recognition.Although current methods trained using single-view image...With the popularity of the digital human body,monocular three-dimensional(3D)face reconstruction is widely used in fields such as animation and face recognition.Although current methods trained using single-view image sets perform well in monocular 3D face reconstruction tasks,they tend to rely on the constraints of the a priori model or the appearance conditions of the input images,fundamentally because of the inability to propose an effective method to reduce the effects of two-dimensional(2D)ambiguity.To solve this problem,we developed an unsupervised training framework for monocular face 3D reconstruction using rotational cycle consistency.Specifically,to learn more accurate facial information,we first used an autoencoder to factor the input images and applied these factors to generate normalized frontal views.We then proceeded through a differentiable renderer to use rotational consistency to continuously perceive refinement.Our method provided implicit multi-view consistency constraints on the pose and depth information estimation of the input face,and the performance was accurate and robust in the presence of large variations in expression and pose.In the benchmark tests,our method performed more stably and realistically than other methods that used 3D face reconstruction in monocular 2D images.展开更多
In this paper, we proposed a combined PCA-LPP algorithm toimprove 3D face reconstruction performance. Principal component analysis(PCA) is commonly used to compress images and extract features. Onedisadvantage of PCA ...In this paper, we proposed a combined PCA-LPP algorithm toimprove 3D face reconstruction performance. Principal component analysis(PCA) is commonly used to compress images and extract features. Onedisadvantage of PCA is local feature loss. To address this, various studies haveproposed combining a PCA-LPP-based algorithm with a locality preservingprojection (LPP). However, the existing PCA-LPP method is unsuitable for3D face reconstruction because it focuses on data classification and clustering.In the existing PCA-LPP, the adjacency graph, which primarily shows the connectionrelationships between data, is composed of the e-or k-nearest neighbortechniques. By contrast, in this study, complex and detailed parts, such aswrinkles around the eyes and mouth, can be reconstructed by composing thetopology of the 3D face model as an adjacency graph and extracting localfeatures from the connection relationship between the 3D model vertices.Experiments verified the effectiveness of the proposed method. When theproposed method was applied to the 3D face reconstruction evaluation set,a performance improvement of 10% to 20% was observed compared with theexisting PCA-based method.展开更多
In this paper, we propose a highly automatic approach for 3D photorealistic face reconstruction from a single frontal image. The key point of our work is the implementation of adaptive manifold learning approach. Befo...In this paper, we propose a highly automatic approach for 3D photorealistic face reconstruction from a single frontal image. The key point of our work is the implementation of adaptive manifold learning approach. Beforehand, an active appearance model (AAM) is trained for automatic feature extraction and adaptive locally linear embedding (ALLE) algorithm is utilized to reduce the dimensionality of the 3D database. Then, given an input frontal face image, the corresponding weights between 3D samples and the image are synthesized adaptively according to the AAM selected facial features. Finally, geometry reconstruction is achieved by linear weighted combination of adaptively selected samples. Radial basis function (RBF) is adopted to map facial texture from the frontal image to the reconstructed face geometry. The texture of invisible regions between the face and the ears is interpolated by sampling from the frontal image. This approach has several advantages: (1) Only a single frontal face image is needed for highly automatic face reconstruction; (2) Compared with former works, our reconstruction approach provides higher accuracy; (3) Constraint based RBF texture mapping provides natural appearance for reconstructed face.展开更多
In this paper, we present a new technique of 3D face reconstruction from a sequence of images taken with cameras having varying parameters without the need to grid. This method is based on the estimation of the projec...In this paper, we present a new technique of 3D face reconstruction from a sequence of images taken with cameras having varying parameters without the need to grid. This method is based on the estimation of the projection matrices of the cameras from a symmetry property which characterizes the face, these projections matrices are used with points matching in each pair of images to determine the 3D points cloud, subsequently, 3D mesh of the face is constructed with 3D Crust algorithm. Lastly, the 2D image is projected on the 3D model to generate the texture mapping. The strong point of the proposed approach is to minimize the constraints of the calibration system： we calibrated the cameras from a symmetry property which characterizes the face, this property gives us the opportunity to know some points of 3D face in a specific well-chosen global reference, to formulate a system of linear and nonlinear equations according to these 3D points, their projection in the image plan and the elements of the projections matrix. Then to solve these equations, we use a genetic algorithm which consists of finding the global optimum without the need of the initial estimation and allows to avoid the local minima of the formulated cost function. Our study is conducted on real data to demonstrate the validity and the performance of the proposed approach in terms of robustness, simplicity, stability and convergence.展开更多
Research on reconstructing imperfect faces is a challenging task.In this study,we explore a data-driven approach using a pre-trained MICA(MetrIC fAce)model combined with 3D printing to address this challenge.We propos...Research on reconstructing imperfect faces is a challenging task.In this study,we explore a data-driven approach using a pre-trained MICA(MetrIC fAce)model combined with 3D printing to address this challenge.We propose a training strategy that utilizes the pre-trained MICA model and self-supervised learning techniques to improve accuracy and reduce the time needed for 3D facial structure reconstruction.Our results demonstrate high accuracy,evaluated by the geometric loss function and various statistical measures.To showcase the effectiveness of the approach,we used 3D printing to create a model that covers facial wounds.The findings indicate that our method produces a model that fits well and achieves comprehensive 3D facial reconstruction.This technique has the potential to aid doctors in treating patients with facial injuries.展开更多
Cascaded regression has been recently applied to reconstruct 3D faces from single 2D images directly in shape space, and has achieved state-of-the-art performance. We investigate thoroughly such cascaded regression ba...Cascaded regression has been recently applied to reconstruct 3D faces from single 2D images directly in shape space, and has achieved state-of-the-art performance. We investigate thoroughly such cascaded regression based 3D face reconstruction approaches from four perspectives that are not well been studied： （1） the impact of the number of 2D landmarks; （2） the impact of the number of 3D vertices; （3） the way of using standalone automated landmark detection methods; （4） the convergence property. To answer these questions, a simplified cascaded regression based 3D face reconstruction method is devised. This can be integrated with standalone automated landmark detection methods and reconstruct 3D face shapes that have the same pose and expression as the input face images, rather than normalized pose and expression. An effective training method is also proposed by disturbing the automatically detected landmarks. Comprehensive evaluation experiments have been carried out to compare to other 3D face reconstruction methods. The results not only deepen the understanding of cascaded regression based 3D face reconstruction approaches, but also prove the effectiveness of the proposed method.展开更多
Facial wound segmentation plays a crucial role in preoperative planning and optimizing patient outcomes in various medical applications.In this paper,we propose an efficient approach for automating 3D facial wound seg...Facial wound segmentation plays a crucial role in preoperative planning and optimizing patient outcomes in various medical applications.In this paper,we propose an efficient approach for automating 3D facial wound segmentation using a two-stream graph convolutional network.Our method leverages the Cir3D-FaIR dataset and addresses the challenge of data imbalance through extensive experimentation with different loss functions.To achieve accurate segmentation,we conducted thorough experiments and selected a high-performing model from the trainedmodels.The selectedmodel demonstrates exceptional segmentation performance for complex 3D facial wounds.Furthermore,based on the segmentation model,we propose an improved approach for extracting 3D facial wound fillers and compare it to the results of the previous study.Our method achieved a remarkable accuracy of 0.9999993% on the test suite,surpassing the performance of the previous method.From this result,we use 3D printing technology to illustrate the shape of the wound filling.The outcomes of this study have significant implications for physicians involved in preoperative planning and intervention design.By automating facial wound segmentation and improving the accuracy ofwound-filling extraction,our approach can assist in carefully assessing and optimizing interventions,leading to enhanced patient outcomes.Additionally,it contributes to advancing facial reconstruction techniques by utilizing machine learning and 3D bioprinting for printing skin tissue implants.Our source code is available at https://github.com/SIMOGroup/WoundFilling3D.展开更多
Background With the development of virtual reality(VR)technology,there is a growing need for customized 3D avatars.However,traditional methods for 3D avatar modeling are either time-consuming or fail to retain the sim...Background With the development of virtual reality(VR)technology,there is a growing need for customized 3D avatars.However,traditional methods for 3D avatar modeling are either time-consuming or fail to retain the similarity to the person being modeled.This study presents a novel framework for generating animatable 3D cartoon faces from a single portrait image.Methods First,we transferred an input real-world portrait to a stylized cartoon image using StyleGAN.We then proposed a two-stage reconstruction method to recover a 3D cartoon face with detailed texture.Our two-stage strategy initially performs coarse estimation based on template models and subsequently refines the model by nonrigid deformation under landmark supervision.Finally,we proposed a semantic-preserving face-rigging method based on manually created templates and deformation transfer.Conclusions Compared with prior arts,the qualitative and quantitative results show that our method achieves better accuracy,aesthetics,and similarity criteria.Furthermore,we demonstrated the capability of the proposed 3D model for real-time facial animation.展开更多
The skewed symmetry detection plays an improtant role in three-dimensional（3-D） reconstruction. The skewed symmetry depicts a real symmetry viewed from some unknown viewing directions. And the skewed symmetry detect...The skewed symmetry detection plays an improtant role in three-dimensional（3-D） reconstruction. The skewed symmetry depicts a real symmetry viewed from some unknown viewing directions. And the skewed symmetry detection can decrease the geometric constrains and the complexity of 3-D reconstruction. The detection technique for the quadric curve ellipse proposed by Sugimoto is improved to further cover quadric curves including hyperbola and parabola. With the parametric detection, the 3-D quadric curve projection matching is automatical- ly accomplished. Finally, the skewed symmetry surface of the quadric surface solid is obtained. Several examples are used to verify the feasibility of the algorithm and satisfying results can be obtained.展开更多
Images with human faces comprise an essential part in the imaging realm. Occlusion or damage in facial portions will bring a remarkable discomfort and information loss. We propose an algorithm that can repair occluded...Images with human faces comprise an essential part in the imaging realm. Occlusion or damage in facial portions will bring a remarkable discomfort and information loss. We propose an algorithm that can repair occluded or damaged facial images automatically, named ‘facial image inpainting＇. Inpainting is a set of image processing methods to recover missing image portions. We extend the image inpainting methods by introducing facial domain knowledge. With the support of a face database, our approach propagates structural information, i.e., feature points and edge maps, from similar faces to the missing facial regions. Using the interred structural information as guidance, an exemplar-based image inpainting algorithm is employed to copy patches in the same face from the source portion to the missing portion. This newly proposed concept of facial image inpainting outperforms the traditional inpainting methods by propagating the facial shapes from a face database, and avoids the problem of variations in imaging conditions from different images by inferring colors and textures from the same face image. Our system produces seamless faces that are hardly seen drawbacks.展开更多
To the editor In 2003,Schwabegger et al.proposed the muscle-sparing latissimus dorsi(MS-LD)flap[1],which preserved a portion of the latissimus dorsi(LD)muscle around the point where the thoracodorsal artery(TDA)perfor...To the editor In 2003,Schwabegger et al.proposed the muscle-sparing latissimus dorsi(MS-LD)flap[1],which preserved a portion of the latissimus dorsi(LD)muscle around the point where the thoracodorsal artery(TDA)perforator penetrates the muscle,having the advantages of a sufficient flap blood supply and reduced donor-site morbidity.However,the traditional MS-LD flap is still too bulky for the reconstruction of defects of the face and neck,and the size of the traditional flap is sometimes not enough to repair large defects.Thus,combining soft tissue expansion with a vascular supercharging technique,we propose a novel design of MS-LD flap,as well as its application strategy.展开更多
On the basis of the assumption that the human face belongs to a linear class, a multiple-deformation model is proposed to recover face shape from a few points on a single 2D image. Compared to the conventional methods...On the basis of the assumption that the human face belongs to a linear class, a multiple-deformation model is proposed to recover face shape from a few points on a single 2D image. Compared to the conventional methods, this study has the following advantages. First, the proposed modified 3D sparse deforming model is a noniterative approach that can compute global translation efficiently and accurately. Subsequently, the overfitting problem can be alleviated based on the proposed multiple deformation model. Finally, by keeping the main features, the texture generated is realistic. The comparison results show that this novel method outperforms the existing methods by using ground truth data and that realistic 3D faces can be recovered efficiently from a single photograph.展开更多
Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks.However,current reconstruction methods often perform improperly in self-occluded regions ...Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks.However,current reconstruction methods often perform improperly in self-occluded regions and can lead to inaccurate correspondences between a 2D input image and a 3D face template,hindering use in real applications.To address these problems,we propose a deep shape reconstruction and texture completion network,SRTC-Net,which jointly reconstructs 3D facial geometry and completes texture with correspondences from a single input face image.In SRTC-Net,we leverage the geometric cues from completed 3D texture to reconstruct detailed structures of 3D shapes.The SRTC-Net pipeline has three stages.The first introduces a correspondence network to identify pixel-wise correspondence between the input 2D image and a 3D template model,and transfers the input 2D image to a U-V texture map.Then we complete the invisible and occluded areas in the U-V texture map using an inpainting network.To get the 3D facial geometries,we predict coarse shape(U-V position maps)from the segmented face from the correspondence network using a shape network,and then refine the 3D coarse shape by regressing the U-V displacement map from the completed U-V texture map in a pixel-to-pixel way.We examine our methods on 3D reconstruction tasks as well as face frontalization and pose invariant face recognition tasks,using both in-the-lab datasets(MICC,MultiPIE)and in-the-wild datasets(CFP).The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture;they outperform or are comparable to the state-of-the-art.展开更多
3D morphable models(3DMMs)are generative models for face shape and appearance.Recent works impose face recognition constraints on 3DMM shape parameters so that the face shapes of the same person remain consistent.Howe...3D morphable models(3DMMs)are generative models for face shape and appearance.Recent works impose face recognition constraints on 3DMM shape parameters so that the face shapes of the same person remain consistent.However,the shape parameters of traditional 3DMMs satisfy the multivariate Gaussian distribution.In contrast,the identity embeddings meet the hypersphere distribution,and this conflict makes it challenging for face reconstruction models to preserve the faithfulness and the shape consistency simultaneously.In other words,recognition loss and reconstruction loss can not decrease jointly due to their conflict distribution.To address this issue,we propose the Sphere Face Model(SFM),a novel 3DMM for monocular face reconstruction,preserving both shape fidelity and identity consistency.The core of our SFM is the basis matrix which can be used to reconstruct 3D face shapes,and the basic matrix is learned by adopting a twostage training approach where 3D and 2D training data are used in the first and second stages,respectively.We design a novel loss to resolve the distribution mismatch,enforcing that the shape parameters have the hyperspherical distribution.Our model accepts 2D and 3D data for constructing the sphere face models.Extensive experiments show that SFM has high representation ability and clustering performance in its shape parameter space.Moreover,it produces highfidelity face shapes consistently in challenging conditions in monocular face reconstruction.The code will be released at https://github.com/a686432/SIR.展开更多
Face views are particularly important in person-to-person communication.Differenes between the camera location and the face orientation can result in undesirable facial appearances of the participants during video con...Face views are particularly important in person-to-person communication.Differenes between the camera location and the face orientation can result in undesirable facial appearances of the participants during video conferencing.This phenomenon is particularly noticeable when using devices where the frontfacing camera is placed in unconventional locations such as below the display or within the keyboard.In this paper,we take a video stream from a single RGB camera as input,and generate a video stream that emulates the view from a virtual camera at a designated location.The most challenging issue in this problem is that the corrected view often needs out-of-plane head rotations.To address this challenge,we reconstruct the 3D face shape and re-render it into synthesized frames according to the virtual camera location.To output the corrected video stream with natural appearance in real time,we propose several novel techniques including accurate eyebrow reconstruction,high-quality blending between the corrected face image and background,and template-based 3D reconstruction of glasses.Our system works well for different lighting conditions and skin tones,and can handle users wearing glasses.Extensive experiments and user studies demonstrate that our method provides high-quality results.展开更多
Recent advancements in the field have resulted in significant progress in achieving realistic head reconstruction and manipulation using neural radiance fields(NeRF).Despite these advances,capturing intricate facial d...Recent advancements in the field have resulted in significant progress in achieving realistic head reconstruction and manipulation using neural radiance fields(NeRF).Despite these advances,capturing intricate facial details remains a persistent challenge.Moreover,casually captured input,involving both head poses and camera movements,introduces additional difficulties to existing methods of head avatar reconstruction.To address the challenge posed by video data captured with camera motion,we propose a novel method,AvatarWild,for reconstructing head avatars from monocular videos taken by consumer devices.Notably,our approach decouples the camera pose and head pose,allowing reconstructed avatars to be visualized with different poses and expressions from novel viewpoints.To enhance the visual quality of the reconstructed facial avatar,we introduce a view-dependent detail enhancement module designed to augment local facial details without compromising viewpoint consistency.Our method demonstrates superior performance compared to existing approaches,as evidenced by reconstruction and animation results on both multi-view and single-view datasets.Remarkably,our approach stands out by exclusively relying on video data captured by portable devices,such as smartphones.This not only underscores the practicality of our method but also extends its applicability to real-world scenarios where accessibility and ease of data capture are crucial.展开更多
Advances in mobile cameras have made it easier to capture ultra-high resolution(UHR)portraits.However,existing face reconstruction methods lack specific adaptations for UHR input(e.g.,4096×4096),leading to under-...Advances in mobile cameras have made it easier to capture ultra-high resolution(UHR)portraits.However,existing face reconstruction methods lack specific adaptations for UHR input(e.g.,4096×4096),leading to under-use of high-frequency details that are crucial for achieving photorealistic rendering.Our method supports 4096×4096 UHR input and utilizes a divide-and-conquer approach for end-to-end 4K albedo,micronormal,and specular texture reconstruction at the original resolution.We employ a two-stage strategy to capture both global distributions and local high-frequency details,effectively mitigating mosaic and seam artifacts common in patch-based prediction.Additionally,we innovatively apply hash encoding to facial U-V coordinates to boost the model’s ability to learn regional high-frequency feature distributions.Our method can be easily incorporated in stateof-the-art facial geometry reconstruction pipelines,significantly improving the texture reconstruction quality,facilitating artistic creation workflows.展开更多
基金Supported by Science and Technology Department Major Innovation Special Fund of Hubei Province of China(2020BAB116)。
文摘With the popularity of the digital human body,monocular three-dimensional(3D)face reconstruction is widely used in fields such as animation and face recognition.Although current methods trained using single-view image sets perform well in monocular 3D face reconstruction tasks,they tend to rely on the constraints of the a priori model or the appearance conditions of the input images,fundamentally because of the inability to propose an effective method to reduce the effects of two-dimensional(2D)ambiguity.To solve this problem,we developed an unsupervised training framework for monocular face 3D reconstruction using rotational cycle consistency.Specifically,to learn more accurate facial information,we first used an autoencoder to factor the input images and applied these factors to generate normalized frontal views.We then proceeded through a differentiable renderer to use rotational consistency to continuously perceive refinement.Our method provided implicit multi-view consistency constraints on the pose and depth information estimation of the input face,and the performance was accurate and robust in the presence of large variations in expression and pose.In the benchmark tests,our method performed more stably and realistically than other methods that used 3D face reconstruction in monocular 2D images.
基金This research was supported by the Basic Science Research Program through the National Research Foundation of Korea(NRF),funded by the Ministry of Education(2021R1I1A3058103).
文摘In this paper, we proposed a combined PCA-LPP algorithm toimprove 3D face reconstruction performance. Principal component analysis(PCA) is commonly used to compress images and extract features. Onedisadvantage of PCA is local feature loss. To address this, various studies haveproposed combining a PCA-LPP-based algorithm with a locality preservingprojection (LPP). However, the existing PCA-LPP method is unsuitable for3D face reconstruction because it focuses on data classification and clustering.In the existing PCA-LPP, the adjacency graph, which primarily shows the connectionrelationships between data, is composed of the e-or k-nearest neighbortechniques. By contrast, in this study, complex and detailed parts, such aswrinkles around the eyes and mouth, can be reconstructed by composing thetopology of the 3D face model as an adjacency graph and extracting localfeatures from the connection relationship between the 3D model vertices.Experiments verified the effectiveness of the proposed method. When theproposed method was applied to the 3D face reconstruction evaluation set,a performance improvement of 10% to 20% was observed compared with theexisting PCA-based method.
基金Project supported by the National Natural Science Foundation of China (Nos. 60533090, 60525108)the National Basic Research Program (973) of China (No. 2002CB312101)+1 种基金the Science and Technology Project of Zhejiang Province, China (Nos. 2005C13032, 2005C11001-05)China-US Million Book Digital Library Project
文摘In this paper, we propose a highly automatic approach for 3D photorealistic face reconstruction from a single frontal image. The key point of our work is the implementation of adaptive manifold learning approach. Beforehand, an active appearance model (AAM) is trained for automatic feature extraction and adaptive locally linear embedding (ALLE) algorithm is utilized to reduce the dimensionality of the 3D database. Then, given an input frontal face image, the corresponding weights between 3D samples and the image are synthesized adaptively according to the AAM selected facial features. Finally, geometry reconstruction is achieved by linear weighted combination of adaptively selected samples. Radial basis function (RBF) is adopted to map facial texture from the frontal image to the reconstructed face geometry. The texture of invisible regions between the face and the ears is interpolated by sampling from the frontal image. This approach has several advantages: (1) Only a single frontal face image is needed for highly automatic face reconstruction; (2) Compared with former works, our reconstruction approach provides higher accuracy; (3) Constraint based RBF texture mapping provides natural appearance for reconstructed face.
文摘In this paper, we present a new technique of 3D face reconstruction from a sequence of images taken with cameras having varying parameters without the need to grid. This method is based on the estimation of the projection matrices of the cameras from a symmetry property which characterizes the face, these projections matrices are used with points matching in each pair of images to determine the 3D points cloud, subsequently, 3D mesh of the face is constructed with 3D Crust algorithm. Lastly, the 2D image is projected on the 3D model to generate the texture mapping. The strong point of the proposed approach is to minimize the constraints of the calibration system： we calibrated the cameras from a symmetry property which characterizes the face, this property gives us the opportunity to know some points of 3D face in a specific well-chosen global reference, to formulate a system of linear and nonlinear equations according to these 3D points, their projection in the image plan and the elements of the projections matrix. Then to solve these equations, we use a genetic algorithm which consists of finding the global optimum without the need of the initial estimation and allows to avoid the local minima of the formulated cost function. Our study is conducted on real data to demonstrate the validity and the performance of the proposed approach in terms of robustness, simplicity, stability and convergence.
文摘Research on reconstructing imperfect faces is a challenging task.In this study,we explore a data-driven approach using a pre-trained MICA(MetrIC fAce)model combined with 3D printing to address this challenge.We propose a training strategy that utilizes the pre-trained MICA model and self-supervised learning techniques to improve accuracy and reduce the time needed for 3D facial structure reconstruction.Our results demonstrate high accuracy,evaluated by the geometric loss function and various statistical measures.To showcase the effectiveness of the approach,we used 3D printing to create a model that covers facial wounds.The findings indicate that our method produces a model that fits well and achieves comprehensive 3D facial reconstruction.This technique has the potential to aid doctors in treating patients with facial injuries.
基金Project supported by the National Key Research and Development Program of China(Nos.2017YFB0802303and 2016YFC0801100)the National Key Scientific Instrument and Equipment Development Projects of China(No.2013YQ49087904)+1 种基金the National Natural Science Foundation of China(No.61773270)the Miaozi Key Project in Science and Technology Innovation Program of Sichuan Province,China(No.2017RZ0016)
文摘Cascaded regression has been recently applied to reconstruct 3D faces from single 2D images directly in shape space, and has achieved state-of-the-art performance. We investigate thoroughly such cascaded regression based 3D face reconstruction approaches from four perspectives that are not well been studied： （1） the impact of the number of 2D landmarks; （2） the impact of the number of 3D vertices; （3） the way of using standalone automated landmark detection methods; （4） the convergence property. To answer these questions, a simplified cascaded regression based 3D face reconstruction method is devised. This can be integrated with standalone automated landmark detection methods and reconstruct 3D face shapes that have the same pose and expression as the input face images, rather than normalized pose and expression. An effective training method is also proposed by disturbing the automatically detected landmarks. Comprehensive evaluation experiments have been carried out to compare to other 3D face reconstruction methods. The results not only deepen the understanding of cascaded regression based 3D face reconstruction approaches, but also prove the effectiveness of the proposed method.
文摘Facial wound segmentation plays a crucial role in preoperative planning and optimizing patient outcomes in various medical applications.In this paper,we propose an efficient approach for automating 3D facial wound segmentation using a two-stream graph convolutional network.Our method leverages the Cir3D-FaIR dataset and addresses the challenge of data imbalance through extensive experimentation with different loss functions.To achieve accurate segmentation,we conducted thorough experiments and selected a high-performing model from the trainedmodels.The selectedmodel demonstrates exceptional segmentation performance for complex 3D facial wounds.Furthermore,based on the segmentation model,we propose an improved approach for extracting 3D facial wound fillers and compare it to the results of the previous study.Our method achieved a remarkable accuracy of 0.9999993% on the test suite,surpassing the performance of the previous method.From this result,we use 3D printing technology to illustrate the shape of the wound filling.The outcomes of this study have significant implications for physicians involved in preoperative planning and intervention design.By automating facial wound segmentation and improving the accuracy ofwound-filling extraction,our approach can assist in carefully assessing and optimizing interventions,leading to enhanced patient outcomes.Additionally,it contributes to advancing facial reconstruction techniques by utilizing machine learning and 3D bioprinting for printing skin tissue implants.Our source code is available at https://github.com/SIMOGroup/WoundFilling3D.
文摘Background With the development of virtual reality(VR)technology,there is a growing need for customized 3D avatars.However,traditional methods for 3D avatar modeling are either time-consuming or fail to retain the similarity to the person being modeled.This study presents a novel framework for generating animatable 3D cartoon faces from a single portrait image.Methods First,we transferred an input real-world portrait to a stylized cartoon image using StyleGAN.We then proposed a two-stage reconstruction method to recover a 3D cartoon face with detailed texture.Our two-stage strategy initially performs coarse estimation based on template models and subsequently refines the model by nonrigid deformation under landmark supervision.Finally,we proposed a semantic-preserving face-rigging method based on manually created templates and deformation transfer.Conclusions Compared with prior arts,the qualitative and quantitative results show that our method achieves better accuracy,aesthetics,and similarity criteria.Furthermore,we demonstrated the capability of the proposed 3D model for real-time facial animation.
基金Supported by the National Natural Science Foundation of China(10377007)~~
文摘The skewed symmetry detection plays an improtant role in three-dimensional（3-D） reconstruction. The skewed symmetry depicts a real symmetry viewed from some unknown viewing directions. And the skewed symmetry detection can decrease the geometric constrains and the complexity of 3-D reconstruction. The detection technique for the quadric curve ellipse proposed by Sugimoto is improved to further cover quadric curves including hyperbola and parabola. With the parametric detection, the 3-D quadric curve projection matching is automatical- ly accomplished. Finally, the skewed symmetry surface of the quadric surface solid is obtained. Several examples are used to verify the feasibility of the algorithm and satisfying results can be obtained.
基金supported by the National Natural Science Foundation of China (No. 60525108)the National Key Technology R & D Program of China (No. 2006BAH11B03-4)
文摘Images with human faces comprise an essential part in the imaging realm. Occlusion or damage in facial portions will bring a remarkable discomfort and information loss. We propose an algorithm that can repair occluded or damaged facial images automatically, named ‘facial image inpainting＇. Inpainting is a set of image processing methods to recover missing image portions. We extend the image inpainting methods by introducing facial domain knowledge. With the support of a face database, our approach propagates structural information, i.e., feature points and edge maps, from similar faces to the missing facial regions. Using the interred structural information as guidance, an exemplar-based image inpainting algorithm is employed to copy patches in the same face from the source portion to the missing portion. This newly proposed concept of facial image inpainting outperforms the traditional inpainting methods by propagating the facial shapes from a face database, and avoids the problem of variations in imaging conditions from different images by inferring colors and textures from the same face image. Our system produces seamless faces that are hardly seen drawbacks.
基金supported by National Natural Science Foundation of China(82072177,82272264)the‘Hengjie’Program of Shanghai Health Youth Talent Reward Foundation.
文摘To the editor In 2003,Schwabegger et al.proposed the muscle-sparing latissimus dorsi(MS-LD)flap[1],which preserved a portion of the latissimus dorsi(LD)muscle around the point where the thoracodorsal artery(TDA)perforator penetrates the muscle,having the advantages of a sufficient flap blood supply and reduced donor-site morbidity.However,the traditional MS-LD flap is still too bulky for the reconstruction of defects of the face and neck,and the size of the traditional flap is sometimes not enough to repair large defects.Thus,combining soft tissue expansion with a vascular supercharging technique,we propose a novel design of MS-LD flap,as well as its application strategy.
基金the Program for New Century Excellent Talents in University（NCET） The National Natural Science Foundation of China+1 种基金Beijing Natural Science Foundation ProgramBeijing Science and Educational Committee Program.
文摘On the basis of the assumption that the human face belongs to a linear class, a multiple-deformation model is proposed to recover face shape from a few points on a single 2D image. Compared to the conventional methods, this study has the following advantages. First, the proposed modified 3D sparse deforming model is a noniterative approach that can compute global translation efficiently and accurately. Subsequently, the overfitting problem can be alleviated based on the proposed multiple deformation model. Finally, by keeping the main features, the texture generated is realistic. The comparison results show that this novel method outperforms the existing methods by using ground truth data and that realistic 3D faces can be recovered efficiently from a single photograph.
基金supported by the National Natural Science Foundation of China(Nos.U1613211 and U1813218)Shenzhen Research Program(Nos.JCYJ20170818164704758 and JCYJ20150925163005055).
文摘Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks.However,current reconstruction methods often perform improperly in self-occluded regions and can lead to inaccurate correspondences between a 2D input image and a 3D face template,hindering use in real applications.To address these problems,we propose a deep shape reconstruction and texture completion network,SRTC-Net,which jointly reconstructs 3D facial geometry and completes texture with correspondences from a single input face image.In SRTC-Net,we leverage the geometric cues from completed 3D texture to reconstruct detailed structures of 3D shapes.The SRTC-Net pipeline has three stages.The first introduces a correspondence network to identify pixel-wise correspondence between the input 2D image and a 3D template model,and transfers the input 2D image to a U-V texture map.Then we complete the invisible and occluded areas in the U-V texture map using an inpainting network.To get the 3D facial geometries,we predict coarse shape(U-V position maps)from the segmented face from the correspondence network using a shape network,and then refine the 3D coarse shape by regressing the U-V displacement map from the completed U-V texture map in a pixel-to-pixel way.We examine our methods on 3D reconstruction tasks as well as face frontalization and pose invariant face recognition tasks,using both in-the-lab datasets(MICC,MultiPIE)and in-the-wild datasets(CFP).The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture;they outperform or are comparable to the state-of-the-art.
基金supported in part by National Natural Science Foundation of China(61972342,61832016)Science and Technology Department of Zhejiang Province(2018C01080)+2 种基金Zhejiang Province Public Welfare Technology Application Research(LGG22F020009)Key Laboratory of Film and TV Media Technology of Zhejiang Province(2020E10015)Teaching Reform Project of Communication University of Zhejiang(jgxm202131).
文摘3D morphable models(3DMMs)are generative models for face shape and appearance.Recent works impose face recognition constraints on 3DMM shape parameters so that the face shapes of the same person remain consistent.However,the shape parameters of traditional 3DMMs satisfy the multivariate Gaussian distribution.In contrast,the identity embeddings meet the hypersphere distribution,and this conflict makes it challenging for face reconstruction models to preserve the faithfulness and the shape consistency simultaneously.In other words,recognition loss and reconstruction loss can not decrease jointly due to their conflict distribution.To address this issue,we propose the Sphere Face Model(SFM),a novel 3DMM for monocular face reconstruction,preserving both shape fidelity and identity consistency.The core of our SFM is the basis matrix which can be used to reconstruct 3D face shapes,and the basic matrix is learned by adopting a twostage training approach where 3D and 2D training data are used in the first and second stages,respectively.We design a novel loss to resolve the distribution mismatch,enforcing that the shape parameters have the hyperspherical distribution.Our model accepts 2D and 3D data for constructing the sphere face models.Extensive experiments show that SFM has high representation ability and clustering performance in its shape parameter space.Moreover,it produces highfidelity face shapes consistently in challenging conditions in monocular face reconstruction.The code will be released at https://github.com/a686432/SIR.
文摘Face views are particularly important in person-to-person communication.Differenes between the camera location and the face orientation can result in undesirable facial appearances of the participants during video conferencing.This phenomenon is particularly noticeable when using devices where the frontfacing camera is placed in unconventional locations such as below the display or within the keyboard.In this paper,we take a video stream from a single RGB camera as input,and generate a video stream that emulates the view from a virtual camera at a designated location.The most challenging issue in this problem is that the corrected view often needs out-of-plane head rotations.To address this challenge,we reconstruct the 3D face shape and re-render it into synthesized frames according to the virtual camera location.To output the corrected video stream with natural appearance in real time,we propose several novel techniques including accurate eyebrow reconstruction,high-quality blending between the corrected face image and background,and template-based 3D reconstruction of glasses.Our system works well for different lighting conditions and skin tones,and can handle users wearing glasses.Extensive experiments and user studies demonstrate that our method provides high-quality results.
基金supported by National Natural Science Foundation of China(No.6247075018 and No.62322210)the Innovation Funding of ICT,CAS(No.E461020)+1 种基金Beijing Munici-pal Natural Science Foundation for Distinguished Young Scholars(No.JQ21013)Beijing Municipal Science and Technology Commission(No.Z231100005923031).
文摘Recent advancements in the field have resulted in significant progress in achieving realistic head reconstruction and manipulation using neural radiance fields(NeRF).Despite these advances,capturing intricate facial details remains a persistent challenge.Moreover,casually captured input,involving both head poses and camera movements,introduces additional difficulties to existing methods of head avatar reconstruction.To address the challenge posed by video data captured with camera motion,we propose a novel method,AvatarWild,for reconstructing head avatars from monocular videos taken by consumer devices.Notably,our approach decouples the camera pose and head pose,allowing reconstructed avatars to be visualized with different poses and expressions from novel viewpoints.To enhance the visual quality of the reconstructed facial avatar,we introduce a view-dependent detail enhancement module designed to augment local facial details without compromising viewpoint consistency.Our method demonstrates superior performance compared to existing approaches,as evidenced by reconstruction and animation results on both multi-view and single-view datasets.Remarkably,our approach stands out by exclusively relying on video data captured by portable devices,such as smartphones.This not only underscores the practicality of our method but also extends its applicability to real-world scenarios where accessibility and ease of data capture are crucial.
基金supported by the National Key R&D Program of China(2024YDLN0011)the Key R&D Program of Zhejiang Province(2023C01039).
文摘Advances in mobile cameras have made it easier to capture ultra-high resolution(UHR)portraits.However,existing face reconstruction methods lack specific adaptations for UHR input(e.g.,4096×4096),leading to under-use of high-frequency details that are crucial for achieving photorealistic rendering.Our method supports 4096×4096 UHR input and utilizes a divide-and-conquer approach for end-to-end 4K albedo,micronormal,and specular texture reconstruction at the original resolution.We employ a two-stage strategy to capture both global distributions and local high-frequency details,effectively mitigating mosaic and seam artifacts common in patch-based prediction.Additionally,we innovatively apply hash encoding to facial U-V coordinates to boost the model’s ability to learn regional high-frequency feature distributions.Our method can be easily incorporated in stateof-the-art facial geometry reconstruction pipelines,significantly improving the texture reconstruction quality,facilitating artistic creation workflows.
