In artificial intelligence(AI)for science,the AI-empowered topology optimization methods have garnered sustained attention from researchers and achieved significant development.In this paper,we introduce the implicit ...In artificial intelligence(AI)for science,the AI-empowered topology optimization methods have garnered sustained attention from researchers and achieved significant development.In this paper,we introduce the implicit neural representation(INR)from AI and the material point method(MPM)from the field of computational mechanics into topology optimization,resulting in a novel differentiable and fully mesh-independent topology optimization framework named MI-TONR,and it is then applied to nonlinear topology optimization(NTO)design.Within MI-TONR,the INR is combined with the topology description function to construct the design model,while implicit MPM is employed for physical response analysis.A skillful integration is achieved between the design model based on the continuous implicit representation field and the analysis model based on the Lagrangian particles.Along with updating parameters of the neural network(i.e.,design variables),the structural topologies iteratively evolve according to the responses analysis results and optimization functions.The computational differentiability is ensured at every step of MI-TONR,enabling sensitivity analysis using automatic differentiation.In addition,we introduce the augmented Lagrangian Method to handle multiple constraints in topology optimization and adopt a learning rate adaptive adjustment scheme to enhance the robustness of the optimization process.Numerical examples demonstrate that MI-TONR can effectively conduct NTO design under large loads without any numerical techniques to mitigate numerical instabilities.Meanwhile,its natural satisfaction with the no-penetration condition facilitates the NTO design of considering contact.The infinite spatial resolution characteristic facilitates the generation of structural topology at multiple resolutions with clear and continuous boundaries.展开更多
Computed Tomography(CT)reconstruction is essential inmedical imaging and other engineering fields.However,blurring of the projection during CT imaging can lead to artifacts in the reconstructed images.Projection blur ...Computed Tomography(CT)reconstruction is essential inmedical imaging and other engineering fields.However,blurring of the projection during CT imaging can lead to artifacts in the reconstructed images.Projection blur combines factors such as larger ray sources,scattering and imaging system vibration.To address the problem,we propose DeblurTomo,a novel self-supervised learning-based deblurring and reconstruction algorithm that efficiently reconstructs sharp CT images from blurry input without needing external data and blur measurement.Specifically,we constructed a coordinate-based implicit neural representation reconstruction network,which can map the coordinates to the attenuation coefficient in the reconstructed space formore convenient ray representation.Then,wemodel the blur as aweighted sumof offset rays and design the RayCorrectionNetwork(RCN)andWeight ProposalNetwork(WPN)to fit these rays and their weights bymulti-view consistency and geometric information,thereby extending 2D deblurring to 3D space.In the training phase,we use the blurry input as the supervision signal to optimize the reconstruction network,the RCN,and the WPN simultaneously.Extensive experiments on the widely used synthetic dataset show that DeblurTomo performs superiorly on the limited-angle and sparse-view in the simulated blurred scenarios.Further experiments on real datasets demonstrate the superiority of our method in practical scenarios.展开更多
We present a novel framework for audio-guided localized image stylization.Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object.However,...We present a novel framework for audio-guided localized image stylization.Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object.However,existing image stylization works have focused on stylizing the entire image using an image or text input.Stylizing a particular part of the image based on audio input is natural but challenging.This work proposes a framework in which a user provides an audio input to localize the target in the input image and another to locally stylize the target object or scene.We first produce a fine localization map using an audio-visual localization network leveraging CLIP embedding space.We then utilize an implicit neural representation(INR)along with the predicted localization map to stylize the target based on sound information.The INR manipulates local pixel values to be semantically consistent with the provided audio input.Our experiments show that the proposed framework outperforms other audio-guided stylization methods.Moreover,we observe that our method constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.展开更多
Sparse view 3D reconstruction has attracted increasing attention with the development of neural implicit 3D representation.Existing methods usually only make use of 2D views,requiring a dense set of input views for ac...Sparse view 3D reconstruction has attracted increasing attention with the development of neural implicit 3D representation.Existing methods usually only make use of 2D views,requiring a dense set of input views for accurate 3D reconstruction.In this paper,we show that accurate 3D reconstruction can be achieved by incorporating geometric priors into neural implicit 3D reconstruction.Our method adopts the signed distance function as the 3D representation,and learns a generalizable 3D surface reconstruction model from sparse views.Specifically,we build a more effective and sparse feature volume from the input views by using corresponding depth maps,which can be provided by depth sensors or directly predicted from the input views.We recover better geometric details by imposing both depth and surface normal constraints in addition to the color loss when training the neural implicit 3D representation.Experiments demonstrate that our method both outperforms state-of-the-art approaches,and achieves good generalizability.展开更多
We present a multiscale deformed implicit surface network(MDISN)to reconstruct 3D objects from single images by adapting the implicit surface of the target object from coarse to fine to the input image.The basic idea ...We present a multiscale deformed implicit surface network(MDISN)to reconstruct 3D objects from single images by adapting the implicit surface of the target object from coarse to fine to the input image.The basic idea is to optimize the implicit surface according to the change of consecutive feature maps from the input image.And with multi-resolution feature maps,the implicit field is refined progressively,such that lower resolutions outline the main object components,and higher resolutions reveal fine-grained geometric details.To better explore the changes in feature maps,we devise a simple field deformation module that receives two consecutive feature maps to refine the implicit field with finer geometric details.Experimental results on both synthetic and real-world datasets demonstrate the superiority of the proposed method compared to state-of-the-art methods.展开更多
In this paper, we propose NeuS-PIR, a novel approach for learning relightable neural surfaces using pre-integrated rendering from multi-view image observations. Unlike traditional methods based on NeRFs or discrete me...In this paper, we propose NeuS-PIR, a novel approach for learning relightable neural surfaces using pre-integrated rendering from multi-view image observations. Unlike traditional methods based on NeRFs or discrete mesh representations, our approach employs an implicit neural surface representation to reconstruct high-quality geometry. This representation enables the factorization of the radiance field into two components: a spatially varying material field and an all-frequency lighting model. By jointly optimizing this factorization with a differentiable pre-integrated rendering framework, and material encoding regularization, our method effectively addresses the ambiguity in geometry reconstruction, leading to improved disentanglement and refinement of scene properties. Furthermore, we introduce a technique to distill indirect illumination fields, capturing complex lighting effects such as inter-reflections. As a result, NeuS-PIR enables advanced applications like relighting, which can be seamlessly integrated into modern graphics engines. Extensive qualitative and quantitative experiments on both synthetic and real datasets demonstrate that NeuS-PIR outperforms existing methods across various tasks. Source code is available at https://github.com/Sheldonmao/NeuSPIR.展开更多
Neural fields,also known as coordinatebased multi-layer perceptrons(MLPs),have recently achieved impressive results in representing lowdimensional data.Unlike convolutional neural networks(CNNs),MLPs are globally conn...Neural fields,also known as coordinatebased multi-layer perceptrons(MLPs),have recently achieved impressive results in representing lowdimensional data.Unlike convolutional neural networks(CNNs),MLPs are globally connected and lack local control;adjusting a local region leads to global changes.Therefore,improving local neural fields usually leads to a dilemma:filtering out local artifacts can simultaneously smooth away desired details.Our solution is a new filtering technique that consists of two counteractive operators:a smoothing operator that provides global smoothing for better generalization and a recovery operator that provides better controllability for local adjustments.We found that using either operator alone could lead to an increase in noisy artifacts or oversmoothed regions.By combining the two operators,smoothing and sharpening can be adjusted to first smooth the entire region and then recover fine-grained details in the overly smoothed regions.Thus,our filter helps neural fields remove significant noise while enhancing the details.We demonstrate the benefits of our filter on various tasks,where it shows significant improvements over state-of-the-art methods.Moreover,our filter provides a better performance in terms of convergence speed and network stability.展开更多
基金supported by the National Natural Science Foundation of China(Grant No.92371206)the Post-graduate Scientific Research Innovation Project of Hunan Province(Grant No.CX20220059)。
文摘In artificial intelligence(AI)for science,the AI-empowered topology optimization methods have garnered sustained attention from researchers and achieved significant development.In this paper,we introduce the implicit neural representation(INR)from AI and the material point method(MPM)from the field of computational mechanics into topology optimization,resulting in a novel differentiable and fully mesh-independent topology optimization framework named MI-TONR,and it is then applied to nonlinear topology optimization(NTO)design.Within MI-TONR,the INR is combined with the topology description function to construct the design model,while implicit MPM is employed for physical response analysis.A skillful integration is achieved between the design model based on the continuous implicit representation field and the analysis model based on the Lagrangian particles.Along with updating parameters of the neural network(i.e.,design variables),the structural topologies iteratively evolve according to the responses analysis results and optimization functions.The computational differentiability is ensured at every step of MI-TONR,enabling sensitivity analysis using automatic differentiation.In addition,we introduce the augmented Lagrangian Method to handle multiple constraints in topology optimization and adopt a learning rate adaptive adjustment scheme to enhance the robustness of the optimization process.Numerical examples demonstrate that MI-TONR can effectively conduct NTO design under large loads without any numerical techniques to mitigate numerical instabilities.Meanwhile,its natural satisfaction with the no-penetration condition facilitates the NTO design of considering contact.The infinite spatial resolution characteristic facilitates the generation of structural topology at multiple resolutions with clear and continuous boundaries.
基金supported in part by the National Natural Science Foundation of China under Grants 62472434 and 62402171in part by the National Key Research and Development Program of China under Grant 2022YFF1203001+1 种基金in part by the Science and Technology Innovation Program of Hunan Province under Grant 2022RC3061in part by the Sci-Tech Innovation 2030 Agenda under Grant 2023ZD0508600.
文摘Computed Tomography(CT)reconstruction is essential inmedical imaging and other engineering fields.However,blurring of the projection during CT imaging can lead to artifacts in the reconstructed images.Projection blur combines factors such as larger ray sources,scattering and imaging system vibration.To address the problem,we propose DeblurTomo,a novel self-supervised learning-based deblurring and reconstruction algorithm that efficiently reconstructs sharp CT images from blurry input without needing external data and blur measurement.Specifically,we constructed a coordinate-based implicit neural representation reconstruction network,which can map the coordinates to the attenuation coefficient in the reconstructed space formore convenient ray representation.Then,wemodel the blur as aweighted sumof offset rays and design the RayCorrectionNetwork(RCN)andWeight ProposalNetwork(WPN)to fit these rays and their weights bymulti-view consistency and geometric information,thereby extending 2D deblurring to 3D space.In the training phase,we use the blurry input as the supervision signal to optimize the reconstruction network,the RCN,and the WPN simultaneously.Extensive experiments on the widely used synthetic dataset show that DeblurTomo performs superiorly on the limited-angle and sparse-view in the simulated blurred scenarios.Further experiments on real datasets demonstrate the superiority of our method in practical scenarios.
基金supported by the Culture,Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture,Sports and Tourism in 2022-(4D Content Generation and Copyright Protection with Artificial Intelligence,R2022020068,30%Research on neural watermark technology for copyright protection of generative AI 3D content,RS-2024-00348469,40%+1 种基金International Collaborative Research and Global Talent Development for the Development of Copyright Management and Protection Technologies for Generative AI,RS-2024-00345025,10%)the Institute of Information&Communications Technology Planning&Evaluation(IITP)grant funded by the Korea government(MSIT)(RS-2019-II190079,10%,No.2017-0-00417,10%).
文摘We present a novel framework for audio-guided localized image stylization.Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object.However,existing image stylization works have focused on stylizing the entire image using an image or text input.Stylizing a particular part of the image based on audio input is natural but challenging.This work proposes a framework in which a user provides an audio input to localize the target in the input image and another to locally stylize the target object or scene.We first produce a fine localization map using an audio-visual localization network leveraging CLIP embedding space.We then utilize an implicit neural representation(INR)along with the predicted localization map to stylize the target based on sound information.The INR manipulates local pixel values to be semantically consistent with the provided audio input.Our experiments show that the proposed framework outperforms other audio-guided stylization methods.Moreover,we observe that our method constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.
基金supported by the National Natural Science Foundation of China(Grant No.61902210).
文摘Sparse view 3D reconstruction has attracted increasing attention with the development of neural implicit 3D representation.Existing methods usually only make use of 2D views,requiring a dense set of input views for accurate 3D reconstruction.In this paper,we show that accurate 3D reconstruction can be achieved by incorporating geometric priors into neural implicit 3D reconstruction.Our method adopts the signed distance function as the 3D representation,and learns a generalizable 3D surface reconstruction model from sparse views.Specifically,we build a more effective and sparse feature volume from the input views by using corresponding depth maps,which can be provided by depth sensors or directly predicted from the input views.We recover better geometric details by imposing both depth and surface normal constraints in addition to the color loss when training the neural implicit 3D representation.Experiments demonstrate that our method both outperforms state-of-the-art approaches,and achieves good generalizability.
基金This work was supported in part by National Key R&D Program of China(2018YFB1403901,2019YFF0302902)NSF China(61902007)Joint NSFC-ISF Research Grant,China(62161146002).
文摘We present a multiscale deformed implicit surface network(MDISN)to reconstruct 3D objects from single images by adapting the implicit surface of the target object from coarse to fine to the input image.The basic idea is to optimize the implicit surface according to the change of consecutive feature maps from the input image.And with multi-resolution feature maps,the implicit field is refined progressively,such that lower resolutions outline the main object components,and higher resolutions reveal fine-grained geometric details.To better explore the changes in feature maps,we devise a simple field deformation module that receives two consecutive feature maps to refine the implicit field with finer geometric details.Experimental results on both synthetic and real-world datasets demonstrate the superiority of the proposed method compared to state-of-the-art methods.
基金supported by the National Natural Science Foundation of China(62472420).
文摘In this paper, we propose NeuS-PIR, a novel approach for learning relightable neural surfaces using pre-integrated rendering from multi-view image observations. Unlike traditional methods based on NeRFs or discrete mesh representations, our approach employs an implicit neural surface representation to reconstruct high-quality geometry. This representation enables the factorization of the radiance field into two components: a spatially varying material field and an all-frequency lighting model. By jointly optimizing this factorization with a differentiable pre-integrated rendering framework, and material encoding regularization, our method effectively addresses the ambiguity in geometry reconstruction, leading to improved disentanglement and refinement of scene properties. Furthermore, we introduce a technique to distill indirect illumination fields, capturing complex lighting effects such as inter-reflections. As a result, NeuS-PIR enables advanced applications like relighting, which can be seamlessly integrated into modern graphics engines. Extensive qualitative and quantitative experiments on both synthetic and real datasets demonstrate that NeuS-PIR outperforms existing methods across various tasks. Source code is available at https://github.com/Sheldonmao/NeuSPIR.
文摘Neural fields,also known as coordinatebased multi-layer perceptrons(MLPs),have recently achieved impressive results in representing lowdimensional data.Unlike convolutional neural networks(CNNs),MLPs are globally connected and lack local control;adjusting a local region leads to global changes.Therefore,improving local neural fields usually leads to a dilemma:filtering out local artifacts can simultaneously smooth away desired details.Our solution is a new filtering technique that consists of two counteractive operators:a smoothing operator that provides global smoothing for better generalization and a recovery operator that provides better controllability for local adjustments.We found that using either operator alone could lead to an increase in noisy artifacts or oversmoothed regions.By combining the two operators,smoothing and sharpening can be adjusted to first smooth the entire region and then recover fine-grained details in the overly smoothed regions.Thus,our filter helps neural fields remove significant noise while enhancing the details.We demonstrate the benefits of our filter on various tasks,where it shows significant improvements over state-of-the-art methods.Moreover,our filter provides a better performance in terms of convergence speed and network stability.
