One-shot face reenactment is a challenging task due to the identity mismatch between source and driving faces.Most existing methods fail to completely eliminate the interference of driving subjects’identity informati...One-shot face reenactment is a challenging task due to the identity mismatch between source and driving faces.Most existing methods fail to completely eliminate the interference of driving subjects’identity information,which may lead to face shape distortion and undermine the realism of reenactment results.To solve this problem,in this paper,we propose using a 3D morphable model(3DMM)for explicit facial semantic decomposition and identity disentanglement.Instead of using 3D coefficients alone for reenactment control,we take advantage of the generative ability of 3DMM to render textured face proxies.These proxies contain abundant yet compact geometric and semantic information of human faces,which enables us to compute the face motion field between source and driving images by estimating the dense correspondence.In this way,we can approximate reenactment results by warping source images according to the motion field,and a generative adversarial network(GAN)is adopted to further improve the visual quality of warping results.Extensive experiments on various datasets demonstrate the advantages of the proposed method over existing state-of-the-art benchmarks in both identity preservation and reenactment fulfillment.展开更多
In mathematics, physics, and engineering, abstract concepts are an indispensable foundation for the study and comprehension of concrete models. As concepts within these fields become increasingly detached from physica...In mathematics, physics, and engineering, abstract concepts are an indispensable foundation for the study and comprehension of concrete models. As concepts within these fields become increasingly detached from physical entities and more associated with mental events, thinking shifts from analytical to conceptual-abstract. Fundamental topics taken from the abstract algebra (aka: modern algebra) are unquestionably abstract. Historically, fundamental concepts taught from the abstract algebra are detached from physical reality with one exception: Boolean operations. Even so, many abstract algebra texts present Boolean operations from a purely mathematical operator perspective that is detached from physical entities. Some texts on the abstract algebra introduce logic gate circuits, but treat them as perceptual symbols. For majors of pure or applied mathematics, detachments from physical entities is not relevant. For students of Computer and Electrical Engineering (CpE/EE), mental associations of Boolean operations are essential, and one might argue that studying pure Boolean axioms are unnecessary mental abstractions. But by its nature, the CpE/EE field tends to be more mentally abstract than the other engineering disciplines. The depth of the mathematical abstractions that we teach to upper-division CpE/EE majors is certainly up for questioning.展开更多
Games that are used in research on game-based learning and serious games that are used for education are usually exclusive,making it difficult for teachers to attempt new methods in education.According to the data pub...Games that are used in research on game-based learning and serious games that are used for education are usually exclusive,making it difficult for teachers to attempt new methods in education.According to the data published in 2009 by the European Schoolnet(EUN),a network of European Ministries of Education,most of the problems that teachers face when using games in class are related to game selection.Based on the aforementioned data,this study presents a set of criteria for selecting games in order to solve the inconvenience experienced by teachers.The games,which are used as teaching materials for game-based learning,have been replaced with commercial games that are easily purchased,rather than serious games that are difficult to obtain the licensing.Based on the criteria,two games were selected in this study and the classes were conducted at the educational site.Compared to the group trained with textbooks,it has been confirmed that the effect of game-based learning was sufficient.In addition,five out of ten problems that teachers face were resolved.展开更多
Conversation is an essential component of virtual avatar activities in the metaverse.With the development of natural language processing,significant breakthroughs have been made in text and voice conversation generati...Conversation is an essential component of virtual avatar activities in the metaverse.With the development of natural language processing,significant breakthroughs have been made in text and voice conversation generation.However,face-to-face conversations account for the vast majority of daily conversations,while most existing methods focused on single-person talking head generation.In this work,we take a step further and consider generating realistic face-to-face conversation videos.Conversation generation is more challenging than single-person talking head generation,because it requires not only the generation of photo-realistic individual talking heads,but also the listener’s response to the speaker.In this paper,we propose a novel unified framework based on the neural radiance field(NeRF)to address these challenges.Specifically,we model both the speaker and the listener with a NeRF framework under different conditions to control individual expressions.The speaker is driven by the audio signal,while the response of the listener depends on both visual and acoustic information.In this way,face-to-face conversation videos are generated between human avatars,with all the interlocutors modeled within the same network.Moreover,to facilitate future research on this task,we also collected a new human conversation dataset containing 34 video clips.Quantitative and qualitative experiments evaluate our method in different aspects,e.g.,image quality,pose sequence trend,and natural rendering of the scene in the generated videos.Experimental results demonstrate that the avatars in the resulting videos are able to carry on a realistic conversation,and maintain individual styles.展开更多
Recent advancements in the field have resulted in significant progress in achieving realistic head reconstruction and manipulation using neural radiance fields(NeRF).Despite these advances,capturing intricate facial d...Recent advancements in the field have resulted in significant progress in achieving realistic head reconstruction and manipulation using neural radiance fields(NeRF).Despite these advances,capturing intricate facial details remains a persistent challenge.Moreover,casually captured input,involving both head poses and camera movements,introduces additional difficulties to existing methods of head avatar reconstruction.To address the challenge posed by video data captured with camera motion,we propose a novel method,AvatarWild,for reconstructing head avatars from monocular videos taken by consumer devices.Notably,our approach decouples the camera pose and head pose,allowing reconstructed avatars to be visualized with different poses and expressions from novel viewpoints.To enhance the visual quality of the reconstructed facial avatar,we introduce a view-dependent detail enhancement module designed to augment local facial details without compromising viewpoint consistency.Our method demonstrates superior performance compared to existing approaches,as evidenced by reconstruction and animation results on both multi-view and single-view datasets.Remarkably,our approach stands out by exclusively relying on video data captured by portable devices,such as smartphones.This not only underscores the practicality of our method but also extends its applicability to real-world scenarios where accessibility and ease of data capture are crucial.展开更多
基金supported in part by the Beijing Municipal Natural Science Foundation,China(No.4222054)in part by the National Natural Science Foundation of China(Nos.62276263 and 62076240)the Youth Innovation Promotion Association CAS,China(No.Y2023143).
文摘One-shot face reenactment is a challenging task due to the identity mismatch between source and driving faces.Most existing methods fail to completely eliminate the interference of driving subjects’identity information,which may lead to face shape distortion and undermine the realism of reenactment results.To solve this problem,in this paper,we propose using a 3D morphable model(3DMM)for explicit facial semantic decomposition and identity disentanglement.Instead of using 3D coefficients alone for reenactment control,we take advantage of the generative ability of 3DMM to render textured face proxies.These proxies contain abundant yet compact geometric and semantic information of human faces,which enables us to compute the face motion field between source and driving images by estimating the dense correspondence.In this way,we can approximate reenactment results by warping source images according to the motion field,and a generative adversarial network(GAN)is adopted to further improve the visual quality of warping results.Extensive experiments on various datasets demonstrate the advantages of the proposed method over existing state-of-the-art benchmarks in both identity preservation and reenactment fulfillment.
文摘In mathematics, physics, and engineering, abstract concepts are an indispensable foundation for the study and comprehension of concrete models. As concepts within these fields become increasingly detached from physical entities and more associated with mental events, thinking shifts from analytical to conceptual-abstract. Fundamental topics taken from the abstract algebra (aka: modern algebra) are unquestionably abstract. Historically, fundamental concepts taught from the abstract algebra are detached from physical reality with one exception: Boolean operations. Even so, many abstract algebra texts present Boolean operations from a purely mathematical operator perspective that is detached from physical entities. Some texts on the abstract algebra introduce logic gate circuits, but treat them as perceptual symbols. For majors of pure or applied mathematics, detachments from physical entities is not relevant. For students of Computer and Electrical Engineering (CpE/EE), mental associations of Boolean operations are essential, and one might argue that studying pure Boolean axioms are unnecessary mental abstractions. But by its nature, the CpE/EE field tends to be more mentally abstract than the other engineering disciplines. The depth of the mathematical abstractions that we teach to upper-division CpE/EE majors is certainly up for questioning.
基金the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea(2019S1A5C2A04082405)。
文摘Games that are used in research on game-based learning and serious games that are used for education are usually exclusive,making it difficult for teachers to attempt new methods in education.According to the data published in 2009 by the European Schoolnet(EUN),a network of European Ministries of Education,most of the problems that teachers face when using games in class are related to game selection.Based on the aforementioned data,this study presents a set of criteria for selecting games in order to solve the inconvenience experienced by teachers.The games,which are used as teaching materials for game-based learning,have been replaced with commercial games that are easily purchased,rather than serious games that are difficult to obtain the licensing.Based on the criteria,two games were selected in this study and the classes were conducted at the educational site.Compared to the group trained with textbooks,it has been confirmed that the effect of game-based learning was sufficient.In addition,five out of ten problems that teachers face were resolved.
基金supported by the National Natural Science Foundation of China(Nos.62201342 and 62101325)Shanghai Municipal Science and Technology Major Project(No.2021SHZDZX0102).
文摘Conversation is an essential component of virtual avatar activities in the metaverse.With the development of natural language processing,significant breakthroughs have been made in text and voice conversation generation.However,face-to-face conversations account for the vast majority of daily conversations,while most existing methods focused on single-person talking head generation.In this work,we take a step further and consider generating realistic face-to-face conversation videos.Conversation generation is more challenging than single-person talking head generation,because it requires not only the generation of photo-realistic individual talking heads,but also the listener’s response to the speaker.In this paper,we propose a novel unified framework based on the neural radiance field(NeRF)to address these challenges.Specifically,we model both the speaker and the listener with a NeRF framework under different conditions to control individual expressions.The speaker is driven by the audio signal,while the response of the listener depends on both visual and acoustic information.In this way,face-to-face conversation videos are generated between human avatars,with all the interlocutors modeled within the same network.Moreover,to facilitate future research on this task,we also collected a new human conversation dataset containing 34 video clips.Quantitative and qualitative experiments evaluate our method in different aspects,e.g.,image quality,pose sequence trend,and natural rendering of the scene in the generated videos.Experimental results demonstrate that the avatars in the resulting videos are able to carry on a realistic conversation,and maintain individual styles.
基金supported by National Natural Science Foundation of China(No.6247075018 and No.62322210)the Innovation Funding of ICT,CAS(No.E461020)+1 种基金Beijing Munici-pal Natural Science Foundation for Distinguished Young Scholars(No.JQ21013)Beijing Municipal Science and Technology Commission(No.Z231100005923031).
文摘Recent advancements in the field have resulted in significant progress in achieving realistic head reconstruction and manipulation using neural radiance fields(NeRF).Despite these advances,capturing intricate facial details remains a persistent challenge.Moreover,casually captured input,involving both head poses and camera movements,introduces additional difficulties to existing methods of head avatar reconstruction.To address the challenge posed by video data captured with camera motion,we propose a novel method,AvatarWild,for reconstructing head avatars from monocular videos taken by consumer devices.Notably,our approach decouples the camera pose and head pose,allowing reconstructed avatars to be visualized with different poses and expressions from novel viewpoints.To enhance the visual quality of the reconstructed facial avatar,we introduce a view-dependent detail enhancement module designed to augment local facial details without compromising viewpoint consistency.Our method demonstrates superior performance compared to existing approaches,as evidenced by reconstruction and animation results on both multi-view and single-view datasets.Remarkably,our approach stands out by exclusively relying on video data captured by portable devices,such as smartphones.This not only underscores the practicality of our method but also extends its applicability to real-world scenarios where accessibility and ease of data capture are crucial.
