Language-guided fashion image editing is challenging,as fashion image editing is local and requires high precision,while natural language cannot provide precise visual information for guidance.In this paper,we propose...Language-guided fashion image editing is challenging,as fashion image editing is local and requires high precision,while natural language cannot provide precise visual information for guidance.In this paper,we propose LucIE,a novel unsupervised language-guided local image editing method for fashion images.LucIE adopts and modifies recent text-to-image synthesis network,DF-GAN,as its backbone.However,the synthesis backbone often changes the global structure of the input image,making local image editing impractical.To increase structural consistency between input and edited images,we propose Content-Preserving Fusion Module(CPFM).Different from existing fusion modules,CPFM prevents iterative refinement on visual feature maps and accumulates additive modifications on RGB maps.LucIE achieves local image editing explicitly with language-guided image segmentation and maskguided image blending while only using image and text pairs.Results on the DeepFashion dataset shows that LucIE achieves state-of-the-art results.Compared with previous methods,images generated by LucIE also exhibit fewer artifacts.We provide visualizations and perform ablation studies to validate LucIE and the CPFM.We also demonstrate and analyze limitations of LucIE,to provide a better understanding of LucIE.展开更多
To address the limitations of existing instruction-based image editing methods in handling complex Multi-target instructions and maintaining semantic consistency,we present InferEdit,a training-free image editing syst...To address the limitations of existing instruction-based image editing methods in handling complex Multi-target instructions and maintaining semantic consistency,we present InferEdit,a training-free image editing system driven by a Multimodal Large Language Model(MLLM).The system parses complex multi-target instructions into sequential subtasks and performs editing iteratively through target localization and semantic reasoning.Furthermore,to adaptively select the most suitable editing models,we construct the evaluation dataset InferDataset to evaluate various editing models on three types of tasks:object removal,object replacement,and local editing.Based on a comprehensive scoring mechanism,we build Binary Search Trees(BSTs)for different editing types to facilitate model scheduling.Experiments demonstrate that InferEdit outperforms existing methods in handling complex instructions while maintaining semantic consistency and visual quality.展开更多
This paper presents a novel generative model,Collaborative Competitive Agents(CCA),which leverages the capabilities of multiple Large Language Models(LLMs)based agents to execute complex tasks.Drawing inspiration from...This paper presents a novel generative model,Collaborative Competitive Agents(CCA),which leverages the capabilities of multiple Large Language Models(LLMs)based agents to execute complex tasks.Drawing inspiration from Generative Adversarial Networks(GANs),the CCA system employs two equal-status generator agents and a discriminator agent.The generators independently process user instructions and generate results,while the discriminator evaluates the outputs,and provides feedback for the generator agents to further reflect and improve the generation results.Unlike the previous generative model,our system can obtain the intermediate steps of generation.This allows each generator agent to learn from other successful executions due to its transparency,enabling a collaborative competition that enhances the quality and robustness of the system’s results.The primary focus of this study is image editing,demonstrating the CCA’s ability to handle intricate instructions robustly.The paper’s main contributions include the introduction of a multiagent-based generative model with controllable intermediate steps and iterative optimization,a detailed examination of agent relationships,and comprehensive experiments on image editing.展开更多
In this paper we propose a unified variational image editing model. It interprets image editing as a variational problem concerning the adaptive adjustments to the zero- and first-derivatives of the images which corre...In this paper we propose a unified variational image editing model. It interprets image editing as a variational problem concerning the adaptive adjustments to the zero- and first-derivatives of the images which correspond to the color and gradient items. By varying the definition domain of each of the two items as well as applying diverse operators, the new model is capable of tackling a variety of image editing tasks. It achieves visually better seamless image cloning effects than existing approaches. It also induces a new and efficient solution to adjusting the color of an image interactively and locally. Other image editing tasks such as stylized processing, local illumination enhancement and image sharpening, can be accomplished within the unified variational framework. Experimental results verify the high flexibility and efficiency of the proposed model.展开更多
When combining very different images which often contain complex objects and backgrounds,producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general...When combining very different images which often contain complex objects and backgrounds,producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called objectaware image editing, to obtain consistency in structure,color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixeddomain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an objectaware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.展开更多
The ability to quickly and intuitively edit digital content has become increasingly important in our everyday life.However,existing edit propagation methods for editing digital images are typically based on optimizati...The ability to quickly and intuitively edit digital content has become increasingly important in our everyday life.However,existing edit propagation methods for editing digital images are typically based on optimization with high computational cost for large inputs.Moreover,existing edit propagation methods are generally inefficient and highly time-consuming.Accordingly,to improve edit efficiency,this paper proposes a novel edit propagation method using a bilateral grid,which can achieve instant propagation of sparse image edits.Firstly,given an input image with user interactions,we resample each of its pixels into a regularly sampled bilateral grid,which facilitates efficient mapping from an image to the bilateral space.As a result,all pixels with the same feature information(color,coordinates)are clustered to the same grid,which can achieve the goal of reducing both the amount of image data processing and the cost of calculation.We then reformulate the propagation as a function of the interpolation problem in bilateral space,which is solved very efficiently using radial basis functions.Experimental results show that our method improves the efficiency of color editing,making it faster than existing edit approaches,and results in excellent edited images with high quality.展开更多
This paper presents a survey of image synthesis and editing with Generative Adversarial Networks（GANs）. GANs consist of two deep networks, a generator and a discriminator, which are trained in a competitive way. Due...This paper presents a survey of image synthesis and editing with Generative Adversarial Networks（GANs）. GANs consist of two deep networks, a generator and a discriminator, which are trained in a competitive way. Due to the power of deep networks and the competitive training manner, GANs are capable of producing reasonable and realistic images, and have shown great capability in many image synthesis and editing applications.This paper surveys recent GAN papers regarding topics including, but not limited to, texture synthesis, image inpainting, image-to-image translation, and image editing.展开更多
In this paper, we present a new edit tool for the user to conveniently preserve or freely edit the object appearance during seamless image composition. We observe that though Poisson image editing is effective for sea...In this paper, we present a new edit tool for the user to conveniently preserve or freely edit the object appearance during seamless image composition. We observe that though Poisson image editing is effective for seamless image composition. Its color bleeding （the color of the target image is propagated into the source image） is not always desired in applications, and it provides no way to allow the user to edit the appearance of the source image. To make it more flexible and practical, we introduce new energy terms to control the appearance change, and integrate them into the Poisson image editing framework. The new energy function could still be realized using efficient sparse linear solvers, and the user can interactively refine the constraints. With the new tool, the user can enjoy not only seamless image composition, but also the flexibility to preserve or manipulate the appearance of the source image at the same time. This provides more potential for creating new images. Experimental results demonstrate the effectiveness of our new edit tool, with similar time cost to the original Poisson image editing.展开更多
Intrinsic image decomposition decomposes an image into reflectance and shading.It has been applied in image editing,augmented reality,and geometry estimation.However,the complete decoupling between reflectance and sha...Intrinsic image decomposition decomposes an image into reflectance and shading.It has been applied in image editing,augmented reality,and geometry estimation.However,the complete decoupling between reflectance and shading,as well as the consistency of the reconstructed image with the original image,have become the main challenges in the application of intrinsic image decomposition.To improve the performance of the intrinsic image decomposition algorithm for these two challenges,we propose a novel deep learning framework that works separately to learn features unique to different intrinsic images.Based on this framework,we developed more effective loss functions to strengthen the decoupling of reflectance and shading and to maintain the decomposition without losing as much information of the original image as possible.We trained the network on a mixture of synthetic and real datasets and evaluated the results of the experiments on real datasets.The results show that our proposed method not only outperformed existing state-of-the-art methods in qualitative and quantitative comparisons in terms of reflectance but was also competitive in terms of reconstructed consistency and shading.Finally,we implemented several realistic image-editing applications,and the results were visually superior to other results.展开更多
This study introduces a novel conditional recycle generative adversarial network for facial attribute transfor- mation, which can transform high-level semantic face attributes without changing the identity. In our app...This study introduces a novel conditional recycle generative adversarial network for facial attribute transfor- mation, which can transform high-level semantic face attributes without changing the identity. In our approach, we input a source facial image to the conditional generator with target attribute condition to generate a face with the target attribute. Then we recycle the generated face back to the same conditional generator with source attribute condition. A face which should be similar to that of the source face in personal identity and facial attributes is generated. Hence, we introduce a recycle reconstruction loss to enforce the final generated facial image and the source facial image to be identical. Evaluations on the CelebA dataset demonstrate the effectiveness of our approach. Qualitative results show that our approach can learn and generate high-quality identity-preserving facial images with specified attributes.展开更多
Artificial intelligence generated content(AIGC)has emerged as an indispensable tool for producing large-scale content in various forms,such as images,thanks to the significant role that AI plays in imitation and produ...Artificial intelligence generated content(AIGC)has emerged as an indispensable tool for producing large-scale content in various forms,such as images,thanks to the significant role that AI plays in imitation and production.However,interpretability and controllability remain challenges.Existing AI methods often face challenges in producing images that are both flexible and controllable while considering causal relationships within the images.To address this issue,we have developed a novel method for causal controllable image generation(CCIG)that combines causal representation learning with bi-directional generative adversarial networks(GANs).This approach enables humans to control image attributes while considering the rationality and interpretability of the generated images and also allows for the generation of counterfactual images.The key of our approach,CCIG,lies in the use of a causal structure learning module to learn the causal relationships between image attributes and joint optimization with the encoder,generator,and joint discriminator in the image generation module.By doing so,we can learn causal representations in image’s latent space and use causal intervention operations to control image generation.We conduct extensive experiments on a real-world dataset,CelebA.The experimental results illustrate the effectiveness of CCIG.展开更多
In this paper, we present a computationally simple yet effective image recoloring method based on color harmonization. Our method permits the user to obtain recolored results interactively by rotating a harmonious tem...In this paper, we present a computationally simple yet effective image recoloring method based on color harmonization. Our method permits the user to obtain recolored results interactively by rotating a harmonious template after completing color harmonization. Two main improvements are made in this paper. Firstly, we give a new strategy for finding the most harmonious scheme, in terms of finding the template which best matches the hue distribution of the input image. Secondly, in order to achieve spatially coherent harmonization, geodesic distances are used to move hues lying outside the harmonious sectors to inside them. Experiments show that our approach can produce higher-quality visually pleasing recolored images than existing methods. Moreover, our method is simple and easy to implement, and has good runtime performance.展开更多
This study presents anchor-regularized generative adversarial network(GAN)priors to delicately explore the inherent knowledge of a pretrained generative model.Previous research leveraged the latent space of a pretrain...This study presents anchor-regularized generative adversarial network(GAN)priors to delicately explore the inherent knowledge of a pretrained generative model.Previous research leveraged the latent space of a pretrained GAN model to provide a variety of image-editing operations.However,the semantically meaningful regions within latent space are distinctly bounded;therefore,the manipulation of the latent code can easily land out of the domain.To address this problem,we introduce an anchoring mechanism that enables novel and robust image editing.The key insights driving the method are that latent space is structurally organized,and that natural coherence allows semantically correlated latent code to be located in the areas surrounding a meaningful anchor.By using different input anchors,the proposed method forms the basis for a variety of robust and flexible editing operations,including misaligned domain translation,interactive editing,and few-shot interpretable direction exploration.Extensive experiments demonstrated the superior performance of the proposed method compared with state-of-the-art editing methods.展开更多
Color pencil drawing is well-loved due to its rich expressiveness.This paper proposes an approach for generating feature-preserving color pencil drawings from photographs.To mimic the tonal style of color pencil drawi...Color pencil drawing is well-loved due to its rich expressiveness.This paper proposes an approach for generating feature-preserving color pencil drawings from photographs.To mimic the tonal style of color pencil drawings,which are much lighter and have relatively lower saturation than photographs,we devise a lightness enhancement mapping and a saturation reduction mapping.The lightness mapping is a monotonically decreasing derivative function,which not only increases lightness but also preserves input photograph features.Color saturation is usually related to lightness,so we suppress the saturation dependent on lightness to yield a harmonious tone.Finally,two extremum operators are provided to generate a foreground-aware outline map in which the colors of the generated contours and the foreground object are consistent.Comprehensive experiments show that color pencil drawings generated by our method surpass existing methods in tone capture and feature preservation.展开更多
We present a method for transferring lighting between photographs of a static scene. Our method takes as input a photo collection depicting a scene with varying viewpoints and lighting conditions.We cast lighting tran...We present a method for transferring lighting between photographs of a static scene. Our method takes as input a photo collection depicting a scene with varying viewpoints and lighting conditions.We cast lighting transfer as an edit propagation problem, where the transfer of local illumination across images is guided by sparse correspondences obtained through multi-view stereo. Instead of directly propagating color, we learn local color transforms from corresponding patches in pairs of images and propagate these transforms in an edge-aware manner to regions with no correspondences. Our color transforms model the large variability of appearance changes in local regions of the scene, and are robust to missing or inaccurate correspondences. The method is fully automatic and can transfer strong shadows between images. We show applications of our image relighting method for enhancing photographs, browsing photo collections with harmonized lighting, and generating synthetic time-lapse sequences.展开更多
Digital cartoon production requires extensive manual labor to colorize sketches with visually pleasant color composition and color shading.During colorization,the artist usually takes an existing cartoon image as colo...Digital cartoon production requires extensive manual labor to colorize sketches with visually pleasant color composition and color shading.During colorization,the artist usually takes an existing cartoon image as color guidance,particularly when colorizing related characters or an animation sequence.Reference-guided colorization is more intuitive than colorization with other hints,such as color points or scribbles,or text-based hints.Unfortunately,reference-guided colorization is challenging since the style of the colorized image should match the style of the reference image in terms of both global color composition and local color shading.In this paper,we propose a novel learning-based framework which colorizes a sketch based on a color style feature extracted from a reference color image.Our framework contains a color style extractor to extract the color feature from a color image,a colorization network to generate multi-scale output images by combining a sketch and a color feature,and a multi-scale discriminator to improve the reality of the output image.Extensive qualitative and quantitative evaluations show that our method outperforms existing methods,providing both superior visual quality and style reference consistency in the task of reference-based colorization.展开更多
文摘Language-guided fashion image editing is challenging,as fashion image editing is local and requires high precision,while natural language cannot provide precise visual information for guidance.In this paper,we propose LucIE,a novel unsupervised language-guided local image editing method for fashion images.LucIE adopts and modifies recent text-to-image synthesis network,DF-GAN,as its backbone.However,the synthesis backbone often changes the global structure of the input image,making local image editing impractical.To increase structural consistency between input and edited images,we propose Content-Preserving Fusion Module(CPFM).Different from existing fusion modules,CPFM prevents iterative refinement on visual feature maps and accumulates additive modifications on RGB maps.LucIE achieves local image editing explicitly with language-guided image segmentation and maskguided image blending while only using image and text pairs.Results on the DeepFashion dataset shows that LucIE achieves state-of-the-art results.Compared with previous methods,images generated by LucIE also exhibit fewer artifacts.We provide visualizations and perform ablation studies to validate LucIE and the CPFM.We also demonstrate and analyze limitations of LucIE,to provide a better understanding of LucIE.
文摘To address the limitations of existing instruction-based image editing methods in handling complex Multi-target instructions and maintaining semantic consistency,we present InferEdit,a training-free image editing system driven by a Multimodal Large Language Model(MLLM).The system parses complex multi-target instructions into sequential subtasks and performs editing iteratively through target localization and semantic reasoning.Furthermore,to adaptively select the most suitable editing models,we construct the evaluation dataset InferDataset to evaluate various editing models on three types of tasks:object removal,object replacement,and local editing.Based on a comprehensive scoring mechanism,we build Binary Search Trees(BSTs)for different editing types to facilitate model scheduling.Experiments demonstrate that InferEdit outperforms existing methods in handling complex instructions while maintaining semantic consistency and visual quality.
文摘This paper presents a novel generative model,Collaborative Competitive Agents(CCA),which leverages the capabilities of multiple Large Language Models(LLMs)based agents to execute complex tasks.Drawing inspiration from Generative Adversarial Networks(GANs),the CCA system employs two equal-status generator agents and a discriminator agent.The generators independently process user instructions and generate results,while the discriminator evaluates the outputs,and provides feedback for the generator agents to further reflect and improve the generation results.Unlike the previous generative model,our system can obtain the intermediate steps of generation.This allows each generator agent to learn from other successful executions due to its transparency,enabling a collaborative competition that enhances the quality and robustness of the system’s results.The primary focus of this study is image editing,demonstrating the CCA’s ability to handle intricate instructions robustly.The paper’s main contributions include the introduction of a multiagent-based generative model with controllable intermediate steps and iterative optimization,a detailed examination of agent relationships,and comprehensive experiments on image editing.
基金A preliminary version of this paper appeared in Proc. Pacific Graphics 2005, Macao. This work is partially supported by the National Basic Research 973 Program of China （Grant No. 2002CB312100）, the National Natural Science Foundation of China （Grant No. 60403038）, the National Natural Science Foundation of China for Innovative Research Groups （Grant No. 60021201）.
文摘In this paper we propose a unified variational image editing model. It interprets image editing as a variational problem concerning the adaptive adjustments to the zero- and first-derivatives of the images which correspond to the color and gradient items. By varying the definition domain of each of the two items as well as applying diverse operators, the new model is capable of tackling a variety of image editing tasks. It achieves visually better seamless image cloning effects than existing approaches. It also induces a new and efficient solution to adjusting the color of an image interactively and locally. Other image editing tasks such as stylized processing, local illumination enhancement and image sharpening, can be accomplished within the unified variational framework. Experimental results verify the high flexibility and efficiency of the proposed model.
基金supported in part by the National Key Research and Development Plan(Grant No.2016YFC0801005)the National Natural Science Foundation of China(Grant Nos.61772513 and 61402463)the Open Foundation Project of Robot Technology Used for Special Environment Key Laboratory of Sichuan Province in China(Grant No.16kftk01)
文摘When combining very different images which often contain complex objects and backgrounds,producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called objectaware image editing, to obtain consistency in structure,color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixeddomain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an objectaware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.
基金supported by National Natural Science Foundation of China(No.U1836208,No.61402053 and No.61202439)Natural Science Foundation of Hunan Province of China(No.2019JJ50666 and No.2019JJ50655)partly supported by Open Fund of Hunan Key Laboratory of Smart Roadway and Cooperative Vehicle-Infrastructure Systems(Changsha University of Science&Technology)(No.KFJ180701).
文摘The ability to quickly and intuitively edit digital content has become increasingly important in our everyday life.However,existing edit propagation methods for editing digital images are typically based on optimization with high computational cost for large inputs.Moreover,existing edit propagation methods are generally inefficient and highly time-consuming.Accordingly,to improve edit efficiency,this paper proposes a novel edit propagation method using a bilateral grid,which can achieve instant propagation of sparse image edits.Firstly,given an input image with user interactions,we resample each of its pixels into a regularly sampled bilateral grid,which facilitates efficient mapping from an image to the bilateral space.As a result,all pixels with the same feature information(color,coordinates)are clustered to the same grid,which can achieve the goal of reducing both the amount of image data processing and the cost of calculation.We then reformulate the propagation as a function of the interpolation problem in bilateral space,which is solved very efficiently using radial basis functions.Experimental results show that our method improves the efficiency of color editing,making it faster than existing edit approaches,and results in excellent edited images with high quality.
基金supported by the National Key Technology R&D Program(No.2016YFB1001402)the National Natural Science Foundation of China(No.61521002)+2 种基金the Joint NSFC-ISF Research Program(No.61561146393)Research Grant of Beijing Higher Institution Engineering Research Center and Tsinghua-Tencent Joint Laboratory for Internet Innovation Technologysupported by the EPSRC CDE(No.EP/L016540/1)
文摘This paper presents a survey of image synthesis and editing with Generative Adversarial Networks（GANs）. GANs consist of two deep networks, a generator and a discriminator, which are trained in a competitive way. Due to the power of deep networks and the competitive training manner, GANs are capable of producing reasonable and realistic images, and have shown great capability in many image synthesis and editing applications.This paper surveys recent GAN papers regarding topics including, but not limited to, texture synthesis, image inpainting, image-to-image translation, and image editing.
基金supported by the National Natural Science Foundation of China under Grant Nos. 60773026, 60873182,60833007
文摘In this paper, we present a new edit tool for the user to conveniently preserve or freely edit the object appearance during seamless image composition. We observe that though Poisson image editing is effective for seamless image composition. Its color bleeding （the color of the target image is propagated into the source image） is not always desired in applications, and it provides no way to allow the user to edit the appearance of the source image. To make it more flexible and practical, we introduce new energy terms to control the appearance change, and integrate them into the Poisson image editing framework. The new energy function could still be realized using efficient sparse linear solvers, and the user can interactively refine the constraints. With the new tool, the user can enjoy not only seamless image composition, but also the flexibility to preserve or manipulate the appearance of the source image at the same time. This provides more potential for creating new images. Experimental results demonstrate the effectiveness of our new edit tool, with similar time cost to the original Poisson image editing.
基金supported by the Science and Technology Innovation 2030-Major Project of“New Generation Artificial Intelligence”(No.2022ZD0115901)the National Natural Science Foundation of China(No.62332003).
文摘Intrinsic image decomposition decomposes an image into reflectance and shading.It has been applied in image editing,augmented reality,and geometry estimation.However,the complete decoupling between reflectance and shading,as well as the consistency of the reconstructed image with the original image,have become the main challenges in the application of intrinsic image decomposition.To improve the performance of the intrinsic image decomposition algorithm for these two challenges,we propose a novel deep learning framework that works separately to learn features unique to different intrinsic images.Based on this framework,we developed more effective loss functions to strengthen the decoupling of reflectance and shading and to maintain the decomposition without losing as much information of the original image as possible.We trained the network on a mixture of synthetic and real datasets and evaluated the results of the experiments on real datasets.The results show that our proposed method not only outperformed existing state-of-the-art methods in qualitative and quantitative comparisons in terms of reflectance but was also competitive in terms of reconstructed consistency and shading.Finally,we implemented several realistic image-editing applications,and the results were visually superior to other results.
基金This work was supported by the National Natural Science Foundation of China under Grant Nos. 61672520, 61573348, 61620106003, and 61720106006, the Beijing Natural Science Foundation of China under Grant No. 4162056, the National Key Technology Research and Development Program of China under Grant No. 2015BAH53F02, and the CASIA-Tencent YouTu Jointly Research Project. The Titan X used for this research was donated by the NVIDIA Corporation.
文摘This study introduces a novel conditional recycle generative adversarial network for facial attribute transfor- mation, which can transform high-level semantic face attributes without changing the identity. In our approach, we input a source facial image to the conditional generator with target attribute condition to generate a face with the target attribute. Then we recycle the generated face back to the same conditional generator with source attribute condition. A face which should be similar to that of the source face in personal identity and facial attributes is generated. Hence, we introduce a recycle reconstruction loss to enforce the final generated facial image and the source facial image to be identical. Evaluations on the CelebA dataset demonstrate the effectiveness of our approach. Qualitative results show that our approach can learn and generate high-quality identity-preserving facial images with specified attributes.
基金Project supported by the National Major Science and Technology Projects of China(No.2022YFB3303302)the National Natural Science Foundation of China(Nos.61977012 and 62207007)the Central Universities Project in China at Chongqing University(Nos.2021CDJYGRH011 and 2020CDJSK06PT14)。
文摘Artificial intelligence generated content(AIGC)has emerged as an indispensable tool for producing large-scale content in various forms,such as images,thanks to the significant role that AI plays in imitation and production.However,interpretability and controllability remain challenges.Existing AI methods often face challenges in producing images that are both flexible and controllable while considering causal relationships within the images.To address this issue,we have developed a novel method for causal controllable image generation(CCIG)that combines causal representation learning with bi-directional generative adversarial networks(GANs).This approach enables humans to control image attributes while considering the rationality and interpretability of the generated images and also allows for the generation of counterfactual images.The key of our approach,CCIG,lies in the use of a causal structure learning module to learn the causal relationships between image attributes and joint optimization with the encoder,generator,and joint discriminator in the image generation module.By doing so,we can learn causal representations in image’s latent space and use causal intervention operations to control image generation.We conduct extensive experiments on a real-world dataset,CelebA.The experimental results illustrate the effectiveness of CCIG.
基金supported by the National Natural Science Foundation of China (Grant No. 61100146)Zhejiang Provincial Natural Science Foundation of China (Grant Nos. LY15F020019, LQ14F020006, and LQ12F02010)the Science and Technology Plan Program of Wenzhou, China (Grant Nos. G20130017 and S20100053)
文摘In this paper, we present a computationally simple yet effective image recoloring method based on color harmonization. Our method permits the user to obtain recolored results interactively by rotating a harmonious template after completing color harmonization. Two main improvements are made in this paper. Firstly, we give a new strategy for finding the most harmonious scheme, in terms of finding the template which best matches the hue distribution of the input image. Secondly, in order to achieve spatially coherent harmonization, geodesic distances are used to move hues lying outside the harmonious sectors to inside them. Experiments show that our approach can produce higher-quality visually pleasing recolored images than existing methods. Moreover, our method is simple and easy to implement, and has good runtime performance.
基金supported by the National Natural Science Foundation of China(No.61972162)Guangdong Natural Science Foundation(No.2021A1515012625)+1 种基金Guangdong Natural Science Funds for Distinguished Young Scholars(No.2023B1515020097)Singapore Ministry of Education Academic Research Fund Tier 1(No.MSS23C002).
文摘This study presents anchor-regularized generative adversarial network(GAN)priors to delicately explore the inherent knowledge of a pretrained generative model.Previous research leveraged the latent space of a pretrained GAN model to provide a variety of image-editing operations.However,the semantically meaningful regions within latent space are distinctly bounded;therefore,the manipulation of the latent code can easily land out of the domain.To address this problem,we introduce an anchoring mechanism that enables novel and robust image editing.The key insights driving the method are that latent space is structurally organized,and that natural coherence allows semantically correlated latent code to be located in the areas surrounding a meaningful anchor.By using different input anchors,the proposed method forms the basis for a variety of robust and flexible editing operations,including misaligned domain translation,interactive editing,and few-shot interpretable direction exploration.Extensive experiments demonstrated the superior performance of the proposed method compared with state-of-the-art editing methods.
基金This work was supported in parts by GD Natural Science Foundation(2021A1515012301,2022A1515011425)the Key Research and Development Project of Guangzhou(202206010091,SL2022B03J01235).
文摘Color pencil drawing is well-loved due to its rich expressiveness.This paper proposes an approach for generating feature-preserving color pencil drawings from photographs.To mimic the tonal style of color pencil drawings,which are much lighter and have relatively lower saturation than photographs,we devise a lightness enhancement mapping and a saturation reduction mapping.The lightness mapping is a monotonically decreasing derivative function,which not only increases lightness but also preserves input photograph features.Color saturation is usually related to lightness,so we suppress the saturation dependent on lightness to yield a harmonious tone.Finally,two extremum operators are provided to generate a foreground-aware outline map in which the colors of the generated contours and the foreground object are consistent.Comprehensive experiments show that color pencil drawings generated by our method surpass existing methods in tone capture and feature preservation.
基金the National University of Singapore with support from the School of Computingsupported by the Being There Centre,a collaboration between Nanyang Technological University Singapore,Eidgenossische Technische Hochschule Zürich+2 种基金the University of North Carolina at Chapel Hillsupported by the Singapore National Research Foundation under its International Research Centre@Singapore Funding Initiativethe Interactive Digital Media Programme Office
文摘We present a method for transferring lighting between photographs of a static scene. Our method takes as input a photo collection depicting a scene with varying viewpoints and lighting conditions.We cast lighting transfer as an edit propagation problem, where the transfer of local illumination across images is guided by sparse correspondences obtained through multi-view stereo. Instead of directly propagating color, we learn local color transforms from corresponding patches in pairs of images and propagate these transforms in an edge-aware manner to regions with no correspondences. Our color transforms model the large variability of appearance changes in local regions of the scene, and are robust to missing or inaccurate correspondences. The method is fully automatic and can transfer strong shadows between images. We show applications of our image relighting method for enhancing photographs, browsing photo collections with harmonized lighting, and generating synthetic time-lapse sequences.
基金supported in part by a CIHE Institutional Development Grant No.IDG200107the National Natural Science Foundation of China under Grant No.61973221the Natural Science Foundation of Guangdong Province of China under Grant Nos.2018A030313381 and 2019A1515011165.
文摘Digital cartoon production requires extensive manual labor to colorize sketches with visually pleasant color composition and color shading.During colorization,the artist usually takes an existing cartoon image as color guidance,particularly when colorizing related characters or an animation sequence.Reference-guided colorization is more intuitive than colorization with other hints,such as color points or scribbles,or text-based hints.Unfortunately,reference-guided colorization is challenging since the style of the colorized image should match the style of the reference image in terms of both global color composition and local color shading.In this paper,we propose a novel learning-based framework which colorizes a sketch based on a color style feature extracted from a reference color image.Our framework contains a color style extractor to extract the color feature from a color image,a colorization network to generate multi-scale output images by combining a sketch and a color feature,and a multi-scale discriminator to improve the reality of the output image.Extensive qualitative and quantitative evaluations show that our method outperforms existing methods,providing both superior visual quality and style reference consistency in the task of reference-based colorization.
