Background The accurate(quantitative)analysis of 3D face deformation is a problem of increasing interest in many applications.In particular,defining a 3D model of the face deformation into a 2D target image to capture...Background The accurate(quantitative)analysis of 3D face deformation is a problem of increasing interest in many applications.In particular,defining a 3D model of the face deformation into a 2D target image to capture local and asymmetric deformations remains a challenge in existing literature.A measure of such local deformations may be a relevant index for monitoring the rehabilitation exercises of patients suffering from Par-kinson’s or Alzheimer’s disease or those recovering from a stroke.Methods In this paper,a complete framework that allows the construction of a 3D morphable shape model(3DMM)of the face is presented for fitting to a target RGB image.The model has the specific characteristic of being based on localized components of deformation.The fitting transformation is performed from 3D to 2D and guided by the correspondence between landmarks detected in the target image and those manually annotated on the average 3DMM.The fitting also has the distinction of being performed in two steps to disentangle face deformations related to the identity of the target subject from those induced by facial actions.Results The method was experimentally validated using the MICC-3D dataset,which includes 11 subjects.Each subject was imaged in one neutral pose and while performing 18 facial actions that deform the face in localized and asymmetric ways.For each acquisition,3DMM was fit to an RGB frame whereby,from the apex facial action and the neutral frame,the extent of the deformation was computed.The results indicate that the proposed approach can accurately capture face deformation,even localized and asymmetric deformations.Conclusion The proposed framework demonstrated that it is possible to measure deformations of a reconstructed 3D face model to monitor facial actions performed in response to a set of targets.Interestingly,these results were obtained using only RGB targets,without the need for 3D scans captured with costly devices.This paves the way for the use of the proposed tool in remote medical rehabilitation monitoring.展开更多
人脸重演技术作为可控人脸生成领域的关键研究方向,其目标在于通过给定的驱动人脸图像或视频帧,驱动源人脸图像,实现其面部表情和姿态的准确可控合成。该技术要求生成结果既能保持源人脸图像的身份特征,又能与驱动人脸图像的表情姿态保...人脸重演技术作为可控人脸生成领域的关键研究方向,其目标在于通过给定的驱动人脸图像或视频帧,驱动源人脸图像,实现其面部表情和姿态的准确可控合成。该技术要求生成结果既能保持源人脸图像的身份特征,又能与驱动人脸图像的表情姿态保持高度一致。单样本人脸重演任务由于仅依赖单一视角的2D人脸图像,导致面部信息描述不充分。现有方法在生成姿态变化幅度较大的人脸图像时,难以准确地保持人脸身份、表情姿态的一致性。针对该问题,提出了一种基于3D可解释性神经渲染的单样本人脸重演(3D Explainable Neural Rendering Based Single-sample Face Reenactment,3D-ENS)方法。该方法在神经网络内部显式建模出固定的3D人脸结构及纹理信息用于整个重演视频生成阶段,以保证重演结果中人脸身份的一致性和表情姿态变化的稳定性。在此基础上构建了一种神经纹理补全网络,通过多尺度特征学习机制实现高质量面部纹理重建;提出了一种背景运动估计网络,预测驱动后人脸图像的背景,将背景信息与补全后的面部神经纹理渲染(Neural Texture Rendering,NTR)结果进行融合。使用关键点检测模型提供面部一致性约束,进一步提升模型的表观一致性。在主流基准数据集与真实场景数据上的实验证明,所提方法具有较好的身份保持度,能够有效应对面部姿态变化的复杂场景,为人脸重演任务提供了新的解决方案。展开更多
文摘Background The accurate(quantitative)analysis of 3D face deformation is a problem of increasing interest in many applications.In particular,defining a 3D model of the face deformation into a 2D target image to capture local and asymmetric deformations remains a challenge in existing literature.A measure of such local deformations may be a relevant index for monitoring the rehabilitation exercises of patients suffering from Par-kinson’s or Alzheimer’s disease or those recovering from a stroke.Methods In this paper,a complete framework that allows the construction of a 3D morphable shape model(3DMM)of the face is presented for fitting to a target RGB image.The model has the specific characteristic of being based on localized components of deformation.The fitting transformation is performed from 3D to 2D and guided by the correspondence between landmarks detected in the target image and those manually annotated on the average 3DMM.The fitting also has the distinction of being performed in two steps to disentangle face deformations related to the identity of the target subject from those induced by facial actions.Results The method was experimentally validated using the MICC-3D dataset,which includes 11 subjects.Each subject was imaged in one neutral pose and while performing 18 facial actions that deform the face in localized and asymmetric ways.For each acquisition,3DMM was fit to an RGB frame whereby,from the apex facial action and the neutral frame,the extent of the deformation was computed.The results indicate that the proposed approach can accurately capture face deformation,even localized and asymmetric deformations.Conclusion The proposed framework demonstrated that it is possible to measure deformations of a reconstructed 3D face model to monitor facial actions performed in response to a set of targets.Interestingly,these results were obtained using only RGB targets,without the need for 3D scans captured with costly devices.This paves the way for the use of the proposed tool in remote medical rehabilitation monitoring.
文摘人脸重演技术作为可控人脸生成领域的关键研究方向,其目标在于通过给定的驱动人脸图像或视频帧,驱动源人脸图像,实现其面部表情和姿态的准确可控合成。该技术要求生成结果既能保持源人脸图像的身份特征,又能与驱动人脸图像的表情姿态保持高度一致。单样本人脸重演任务由于仅依赖单一视角的2D人脸图像,导致面部信息描述不充分。现有方法在生成姿态变化幅度较大的人脸图像时,难以准确地保持人脸身份、表情姿态的一致性。针对该问题,提出了一种基于3D可解释性神经渲染的单样本人脸重演(3D Explainable Neural Rendering Based Single-sample Face Reenactment,3D-ENS)方法。该方法在神经网络内部显式建模出固定的3D人脸结构及纹理信息用于整个重演视频生成阶段,以保证重演结果中人脸身份的一致性和表情姿态变化的稳定性。在此基础上构建了一种神经纹理补全网络,通过多尺度特征学习机制实现高质量面部纹理重建;提出了一种背景运动估计网络,预测驱动后人脸图像的背景,将背景信息与补全后的面部神经纹理渲染(Neural Texture Rendering,NTR)结果进行融合。使用关键点检测模型提供面部一致性约束,进一步提升模型的表观一致性。在主流基准数据集与真实场景数据上的实验证明,所提方法具有较好的身份保持度,能够有效应对面部姿态变化的复杂场景,为人脸重演任务提供了新的解决方案。
