【目的】为应对当前语音驱动的人脸图像生成方法在特征提取和生成质量方面的挑战,特别是解决音频与人脸特征之间深层联系的探索和利用不足问题,提出了一种基于梅尔频率倒谱系数(MFCC)的InceptionResNet-V1音频特征提取网络。【方法】通...【目的】为应对当前语音驱动的人脸图像生成方法在特征提取和生成质量方面的挑战,特别是解决音频与人脸特征之间深层联系的探索和利用不足问题,提出了一种基于梅尔频率倒谱系数(MFCC)的InceptionResNet-V1音频特征提取网络。【方法】通过SEGAN对音频信号进行数据增强,以实现特征的精细提取和有效传递。针对人脸图像生成质量问题,采用基于辅助分类器的生成对抗网络(AC-GAN)作为基线模型,并引入中值增强空间通道注意力模块(Median-enhancedSpatial and Channel Attention Block)以提升特征提取能力。同时,结合图像超分辨率重建模块,将生成的图像恢复为高分辨率图像。【结果】实验结果表明,所提方法在语音驱动的人脸图像生成任务中显著提升了生成质量,相较于主流模型FID降低了36%,余弦相似度提高了22%,人脸检索性能(Top-N)均有效提升,充分证明了其有效性和优越性。【结论】通过语音特征优化和注意力增强机制,有效提升了语音驱动人脸生成的精度与视觉效果,为跨模态生成任务提供了可扩展的技术路径。展开更多
In this paper,we present an efficient algorithm that generates lip-synchronized facial animation from a given vocal audio clip.By combining spectral-dimensional bidirectional long short-term memory and temporal attent...In this paper,we present an efficient algorithm that generates lip-synchronized facial animation from a given vocal audio clip.By combining spectral-dimensional bidirectional long short-term memory and temporal attention mechanism,we design a light-weight speech encoder that leams useful and robust vocal features from the input audio without resorting to pre-trained speech recognition modules or large training data.To learn subject-independent facial motion,we use deformation gradients as the internal representation,which allows nuanced local motions to be better synthesized than using vertex offsets.Compared with state-of-the-art automatic-speech-recognition-based methods,our model is much smaller but achieves similar robustness and quality most of the time,and noticeably better results in certain challenging cases.展开更多
文摘【目的】为应对当前语音驱动的人脸图像生成方法在特征提取和生成质量方面的挑战,特别是解决音频与人脸特征之间深层联系的探索和利用不足问题,提出了一种基于梅尔频率倒谱系数(MFCC)的InceptionResNet-V1音频特征提取网络。【方法】通过SEGAN对音频信号进行数据增强,以实现特征的精细提取和有效传递。针对人脸图像生成质量问题,采用基于辅助分类器的生成对抗网络(AC-GAN)作为基线模型,并引入中值增强空间通道注意力模块(Median-enhancedSpatial and Channel Attention Block)以提升特征提取能力。同时,结合图像超分辨率重建模块,将生成的图像恢复为高分辨率图像。【结果】实验结果表明,所提方法在语音驱动的人脸图像生成任务中显著提升了生成质量,相较于主流模型FID降低了36%,余弦相似度提高了22%,人脸检索性能(Top-N)均有效提升,充分证明了其有效性和优越性。【结论】通过语音特征优化和注意力增强机制,有效提升了语音驱动人脸生成的精度与视觉效果,为跨模态生成任务提供了可扩展的技术路径。
文摘In this paper,we present an efficient algorithm that generates lip-synchronized facial animation from a given vocal audio clip.By combining spectral-dimensional bidirectional long short-term memory and temporal attention mechanism,we design a light-weight speech encoder that leams useful and robust vocal features from the input audio without resorting to pre-trained speech recognition modules or large training data.To learn subject-independent facial motion,we use deformation gradients as the internal representation,which allows nuanced local motions to be better synthesized than using vertex offsets.Compared with state-of-the-art automatic-speech-recognition-based methods,our model is much smaller but achieves similar robustness and quality most of the time,and noticeably better results in certain challenging cases.
