While neural radiance field(NeRF)methods have shown promising results in generating talking faces,existing studies primarily focus on the correlation between avatars and driving sources.However,these studies often ove...While neural radiance field(NeRF)methods have shown promising results in generating talking faces,existing studies primarily focus on the correlation between avatars and driving sources.However,these studies often overlook emotion modeling,resulting in the generation of emotionless or unnatural facial animations.In response,this paper introduces an audio-driven and emotion-editing dynamic NeRF(AED-NeRF)approach,designed for the real-time generation of expressive talking face avatars driven by audio inputs.Specifically,we integrate audio features into a grid-based NeRF to compensate for the lack of a deformation channel,successfully capturing lip dynamics and enabling end-to-end generation from audio-driven sources to talking face avatars.Emotion labels,comprising emotion categories and intensity levels,guide the proposed NeRF framework to implicitly model visual emotions,allowing for explicit control and editing of facial expressions.Extensive qualitative and quantitative experiments validate the effectiveness and advantages of our proposed method,demonstrating its ability to achieve real-time,photo-realistic talking face avatar generation across different audio and emotion scenarios.展开更多
基金supported by ZTE Industry-University-Institute Cooperation Funds under Grant No.IA20230921015。
文摘While neural radiance field(NeRF)methods have shown promising results in generating talking faces,existing studies primarily focus on the correlation between avatars and driving sources.However,these studies often overlook emotion modeling,resulting in the generation of emotionless or unnatural facial animations.In response,this paper introduces an audio-driven and emotion-editing dynamic NeRF(AED-NeRF)approach,designed for the real-time generation of expressive talking face avatars driven by audio inputs.Specifically,we integrate audio features into a grid-based NeRF to compensate for the lack of a deformation channel,successfully capturing lip dynamics and enabling end-to-end generation from audio-driven sources to talking face avatars.Emotion labels,comprising emotion categories and intensity levels,guide the proposed NeRF framework to implicitly model visual emotions,allowing for explicit control and editing of facial expressions.Extensive qualitative and quantitative experiments validate the effectiveness and advantages of our proposed method,demonstrating its ability to achieve real-time,photo-realistic talking face avatar generation across different audio and emotion scenarios.
