Recent advancements in artificial intelligence have transformed three-dimensional(3D)optical imaging and metrology,enabling high-resolution and high-precision 3D surface geometry measurements from one single fringe pa...Recent advancements in artificial intelligence have transformed three-dimensional(3D)optical imaging and metrology,enabling high-resolution and high-precision 3D surface geometry measurements from one single fringe pattern projection.However,the imaging speed of conventional fringe projection profilometry(FPP)remains limited by the native sensor refresh rates due to the inherent"one-to-one"synchronization mechanism between pattern projection and image acquisition in standard structured light techniques.Here,we present dual-frequency angular-multiplexed fringe projection profilometry(DFAMFPP),a deep learning-enabled 3D imaging technique that achieves high-speed,high-precision,and large-depth-range absolute 3D surface measurements at speeds 16 times faster than the sensor's native frame rate.By encoding multi-timeframe 3D information into a single multiplexed image using multiple pairs of dual-frequency fringes,high-accuracy absolute phase maps are reconstructed using specially trained two-stage number-theoretical-based deep neural networks.We validate the effectiveness of DFAMFPP through dynamic scene measurements,achieving 10,000 Hz 3D imaging of a running turbofan engine prototype with only a 625 Hz camera.By overcoming the sensor hardware bottleneck,DFAMFPP significantly advances high-speed and ultra-high-speed 3D imaging,opening new avenues for exploring dynamic processes across diverse scientific disciplines.展开更多
Single-shot high-speed 3D imaging is important for reconstructions of dynamic objects.For fringe projection profilometry(FPP),however,it is still challenging to recover accurate 3D shapes of isolated objects by a sing...Single-shot high-speed 3D imaging is important for reconstructions of dynamic objects.For fringe projection profilometry(FPP),however,it is still challenging to recover accurate 3D shapes of isolated objects by a single fringe image.In this paper,we demonstrate that the deep neural networks can be trained to directly recover the absolute phase from a unique fringe image that involves spatially multiplexed fringe patterns of different frequencies.The extracted phase is free from spectrum-aliasing problem which is hard to avoid for traditional spatial-multiplexing methods.Experiments on both static and dynamic scenes show that the proposed approach is robust to object motion and can obtain high-quality 3D reconstructions of isolated objects within a single fringe image.展开更多
Fringe projection profilometry(FPP)is a method that determines height by analyzing distortional fringes,which is widely used in high-accuracy 3D imaging.Now,one major reason limiting imaging speed in FPP is the projec...Fringe projection profilometry(FPP)is a method that determines height by analyzing distortional fringes,which is widely used in high-accuracy 3D imaging.Now,one major reason limiting imaging speed in FPP is the projection device;the capture speed of high-speed cameras far exceeds the projection frequency.Among various devices,an LED array can exceed the speed of a high-speed camera.However,non-sinusoidal fringe patterns in the LED array systems can arise from several factors that will reduce the accuracy,such as the spacing between adjacent LEDs,the inconsistency in brightness across different LEDs,and the residual high-order harmonics in binary defocusing projection.It is challenging to resolve by other methods.In this paper,we propose a method that creates a look-up table using system calibration data of phase-height models.Then we utilize the look-up table to compensate for the phase error during the reconstructing process.The foundation of the proposed method relies on the time-invariance of systematic error;any factor that impacts the sinusoidal characteristic would present as an anomaly in the unwrapped phase.Experiments have demonstrated that the root mean square errors(RMSEs)of the results yielded by the proposed method were reduced by over 90%compared to those yielded by the traditional method,reaching 20μm accuracy.This paper offers an alternative approach for high-speed and high-accuracy 3D imaging with an LED array and presents a workable solution for addressing complex errors from non-sinusoidal fringes.展开更多
Fringe projection profilometry,a powerful technique for three-dimensional(3D)imaging and measurement,has been revolutionized by deep learning,achieving speeds of up to 100,000 frames per second(fps)while preserving hi...Fringe projection profilometry,a powerful technique for three-dimensional(3D)imaging and measurement,has been revolutionized by deep learning,achieving speeds of up to 100,000 frames per second(fps)while preserving highresolution.This advancement expands its applications to high-speed transient scenarios,opening new possibilities for ultrafast 3D measurements.展开更多
Recent advances in imaging sensors and digital light projection technology have facilitated rapid progress in 3D optical sensing,enabling 3D surfaces of complexshaped objects to be captured with high resolution and ac...Recent advances in imaging sensors and digital light projection technology have facilitated rapid progress in 3D optical sensing,enabling 3D surfaces of complexshaped objects to be captured with high resolution and accuracy.Nevertheless,due to the inherent synchronous pattern projection and image acquisition mechanism,the temporal resolution of conventional structured light or fringe projection profilometry(FPP)based 3D imaging methods is still limited to the native detector frame rates.In this work,we demonstrate a new 3D imaging method,termed deep-learning-enabled multiplexed FPP(DLMFPP),that allows to achieve high-resolution and high-speed 3D imaging at near-one-order of magnitude-higher 3D frame rate with conventional low-speed cameras.By encoding temporal information in one multiplexed fringe pattern,DLMFPP harnesses deep neural networks embedded with Fourier transform,phase-shifting and ensemble learning to decompose the pattern and analyze separate fringes,furnishing a high signal-to-noise ratio and a ready-to-implement solution over conventional computational imaging techniques.We demonstrate this method by measuring different types of transient scenes,including rotating fan blades and bullet fired from a toy gun,at kHz using cameras of around 100 Hz.Experiential results establish that DLMFPP allows slow-scan cameras with their known advantages in terms of cost and spatial resolution to be used for high-speed 3D imaging tasks.展开更多
In 2019,the Event Horizon Telescope(EHT)released the first-ever image of a black hole event horizon.Astronomers are now aiming for higher angular resolutions of distant targets,like black holes,to understand more abou...In 2019,the Event Horizon Telescope(EHT)released the first-ever image of a black hole event horizon.Astronomers are now aiming for higher angular resolutions of distant targets,like black holes,to understand more about the fundamental laws of gravity that govern our universe.To achieve this higher resolution and increased sensitivity,larger radio telescopes are needed to operate at higher frequencies and in larger quantities.Projects like the next-generation Very Large Array(ngVLA)and the Square-Kilometer Array(SKA)require building hundreds of telescopes with diameters greater than 10 ms over the next decade.This has a twofold effect.Radio telescope surfaces need to be more accurate to operate at higher frequencies,and the logistics involved in maintaining a radio telescope need to be simplified to support them properly in large quantities.Both of these problems can be solved with improved methods for surface metrology that are faster and more accurate with a higher resolution.This leads to faster and more accurate panel alignment and,therefore,a more productive observatory.In this paper,we present the use of binocular fringe projection profilometry as a solution to this problem and demonstrate it by aligning two panels on a 3-m radio telescope dish.The measurement takes only 10 min and directly delivers feedback on the tip,tilt,and piston of each panel to create the ideal reflector shape.展开更多
Fringe projection profilometry(FPP)has been extensively studied in the field of three-dimensional(3D)measurement.Although FPP always uses high-frequency fringes to ensure high measurement accuracy,too many patterns ar...Fringe projection profilometry(FPP)has been extensively studied in the field of three-dimensional(3D)measurement.Although FPP always uses high-frequency fringes to ensure high measurement accuracy,too many patterns are projected to unwrap the phase,which affects the speed of 3D reconstruction.We propose a high-speed 3D shape measurement method using only three high-frequency inner shifting-phase patterns(70 periods),which satisfies both high precision and high measuring speed requirements.Besides,our proposed method obtains the wrapped phase and the fringe order simultaneously without any other information and constraints.The proposed method has successfully reconstructed moving objects with high speed at the camera's full frame rate(1700 frames per second).展开更多
Fringe projection profilometry(FPP)has been widely applied to non-contact three-dimensional measurement in industries owing to its high accuracy and speed.The point cloud,which is a measurement result of the FPP syste...Fringe projection profilometry(FPP)has been widely applied to non-contact three-dimensional measurement in industries owing to its high accuracy and speed.The point cloud,which is a measurement result of the FPP system,typically contains a large number of invalid points caused by the background,ambient light,shadows,and object edge regions.Research on noisy point detection and elimination has been conducted over the past two decades.However,existing invalid point removal methods are based on image intensity analysis and are only applicable to simple measurement backgrounds that are purely dark.In this paper,we propose a novel invalid point removal framework that consists of two aspects:(1)A convolutional neural network(CNN)is designed to segment the foreground from the background of different intensity conditions in FPP measurement circumstances to remove background points and the most discrete points in background regions.(2)A two-step method based on the fringe image intensity threshold and a bilateral filter is proposed to eliminate the small number of discrete points remaining after background segmentation caused by shadows and edge areas on objects.Experimental results verify that the proposed framework(1)can remove background points intelligently and accurately in different types of complex circumstances,and(2)performs excellently in discrete point detection from object regions.展开更多
The Fringe Projection Profilometry(FPP)system with a single exposure time or a single projection intensity is limited by the dynamic range of the camera,which can lead to overexposure and underexposure of the image,re...The Fringe Projection Profilometry(FPP)system with a single exposure time or a single projection intensity is limited by the dynamic range of the camera,which can lead to overexposure and underexposure of the image,resulting in point cloud loss or reduced accuracy.To address this issue,unlike the pixel modulation method of projectors,we utilize the characteristics of color projectors where the intensity of the three-channel LED can be controlled independently.We propose a method for separating the projector's three-channel light intensity,combined with a color camera,to achieve single exposure and multi-intensity image acquisition.Further,the crosstalk coefficient is applied to predict the three-channel reflectance of the measured object.By integrating clustering and channel mapping,we establish a pixel-level mapping model between the projector's three-channel current and the camera's three-channel image intensity,which realizes the optimal projection current prediction and the high dynamic range(HDR)image acquisition.The proposed method allows for high-precision three-dimensional(3D)data acquisition of HDR scenes with a single exposure.The effectiveness of this method has been validated through experiments with standard planes and standard steps,showing a significant reduction in mean absolute error(44.6%)compared to existing singleexposure HDR methods.Additionally,the number of images required for acquisition is significantly reduced(by 70.8%)compared to multi-exposure fusion methods.This proposed method has great potential in various FPP-related fields.展开更多
In fringe projection profilometry 3D measurement systems,the measurement of surfaces with high variability in reflectivity poses a challenge due to the limited dynamic range of cameras.The main solution involves using...In fringe projection profilometry 3D measurement systems,the measurement of surfaces with high variability in reflectivity poses a challenge due to the limited dynamic range of cameras.The main solution involves using multiple exposures to modulate fringe intensity;however,it is inefficient.In this study,we introduce an attention-guided end-to-end phase calculation network to accelerate the multi-exposure structured light process for high dynamic range(HDR)measurements.We use attention modules to guide feature selection,enhancing relevant features and suppressing irrelevant features.Using the 12-step phase-shifting profilometry(PSP)as ground truth,our method accurately extracts the sine and cosine components of the fundamental frequency from a single pattern to retrieve the absolute phases.Tested on our metallic dataset requiring HDR imaging,our method achieves an absolute phase error of 0.084,close to that of the six-step PSP method(0.069),while using only 16.7%of the time.On the ceramic dataset,our method achieves 0.021 phase error,close to that of the four-step PSP(0.012).In quantitative measurements,our method achieves an accuracy of approximately 40μm on standard spheres and plates.Overall,our method preserves the accuracy of multi-exposure PSP methods while significantly accelerating the 3D reconstruction process.展开更多
基金supported by National Key Research and Development Program of China(2022YFB2804603,2022YFB2804605)National Natural Science Foundation of China(U21B2033)+4 种基金Fundamental Research Funds forthe Central Universities(2023102001,2024202002)National Key Laborato-ry of Shock Wave and Detonation Physics(JCKYS2024212111)China Post-doctoral Science Fund(2023T160318)Open Research Fund of JiangsuKey Laboratory of Spectral Imaging&Intelligent Sense(JSGP202105,JSGP202201)Postgraduate Research&Practice Innovation Program of Jiangsu Province(KYCX25_0695,SJCX25_0188)。
文摘Recent advancements in artificial intelligence have transformed three-dimensional(3D)optical imaging and metrology,enabling high-resolution and high-precision 3D surface geometry measurements from one single fringe pattern projection.However,the imaging speed of conventional fringe projection profilometry(FPP)remains limited by the native sensor refresh rates due to the inherent"one-to-one"synchronization mechanism between pattern projection and image acquisition in standard structured light techniques.Here,we present dual-frequency angular-multiplexed fringe projection profilometry(DFAMFPP),a deep learning-enabled 3D imaging technique that achieves high-speed,high-precision,and large-depth-range absolute 3D surface measurements at speeds 16 times faster than the sensor's native frame rate.By encoding multi-timeframe 3D information into a single multiplexed image using multiple pairs of dual-frequency fringes,high-accuracy absolute phase maps are reconstructed using specially trained two-stage number-theoretical-based deep neural networks.We validate the effectiveness of DFAMFPP through dynamic scene measurements,achieving 10,000 Hz 3D imaging of a running turbofan engine prototype with only a 625 Hz camera.By overcoming the sensor hardware bottleneck,DFAMFPP significantly advances high-speed and ultra-high-speed 3D imaging,opening new avenues for exploring dynamic processes across diverse scientific disciplines.
基金This work was supported by National Natural Science Foundation of China(62075096,62005121,U21B2033)Leading Technology of Jiangsu Basic Research Plan(BK20192003)+4 种基金“333 Engineering”Research Project of Jiangsu Province(BRA2016407)Jiangsu Provincial“One belt and one road”innovation cooperation project(BZ2020007)Fundamental Research Funds for the Central Universities(30921011208,30919011222,30920032101)Postgraduate Research&Practice Innovation Program of Jiangsu Province(KYCX21_0273)Open Research Fund of Jiangsu Key Laboratory of Spectral Imaging&Intelligent Sense(JSGP202105).
文摘Single-shot high-speed 3D imaging is important for reconstructions of dynamic objects.For fringe projection profilometry(FPP),however,it is still challenging to recover accurate 3D shapes of isolated objects by a single fringe image.In this paper,we demonstrate that the deep neural networks can be trained to directly recover the absolute phase from a unique fringe image that involves spatially multiplexed fringe patterns of different frequencies.The extracted phase is free from spectrum-aliasing problem which is hard to avoid for traditional spatial-multiplexing methods.Experiments on both static and dynamic scenes show that the proposed approach is robust to object motion and can obtain high-quality 3D reconstructions of isolated objects within a single fringe image.
基金National Key Research and Development Program of China(2023YFB2806800)Open Research Projects of KLOMT(2022KLOMT02-02)。
文摘Fringe projection profilometry(FPP)is a method that determines height by analyzing distortional fringes,which is widely used in high-accuracy 3D imaging.Now,one major reason limiting imaging speed in FPP is the projection device;the capture speed of high-speed cameras far exceeds the projection frequency.Among various devices,an LED array can exceed the speed of a high-speed camera.However,non-sinusoidal fringe patterns in the LED array systems can arise from several factors that will reduce the accuracy,such as the spacing between adjacent LEDs,the inconsistency in brightness across different LEDs,and the residual high-order harmonics in binary defocusing projection.It is challenging to resolve by other methods.In this paper,we propose a method that creates a look-up table using system calibration data of phase-height models.Then we utilize the look-up table to compensate for the phase error during the reconstructing process.The foundation of the proposed method relies on the time-invariance of systematic error;any factor that impacts the sinusoidal characteristic would present as an anomaly in the unwrapped phase.Experiments have demonstrated that the root mean square errors(RMSEs)of the results yielded by the proposed method were reduced by over 90%compared to those yielded by the traditional method,reaching 20μm accuracy.This paper offers an alternative approach for high-speed and high-accuracy 3D imaging with an LED array and presents a workable solution for addressing complex errors from non-sinusoidal fringes.
文摘Fringe projection profilometry,a powerful technique for three-dimensional(3D)imaging and measurement,has been revolutionized by deep learning,achieving speeds of up to 100,000 frames per second(fps)while preserving highresolution.This advancement expands its applications to high-speed transient scenarios,opening new possibilities for ultrafast 3D measurements.
基金supported by National Key Research and Development Program of China(2022YFB2804603)National Natural Science Foundation of China(62075096,62005121,U21B2033)+3 种基金Leading Technology of Jiangsu Basic Research Plan(BK20192003)“333 Engineering”Research Project of Jiangsu Province(BRA2016407)Fundamental Research Funds for the Central Universities(30921011208,30919011222,30920032101)Fundamental Research Funds for the Central Universities(2023102001,2024202002).
文摘Recent advances in imaging sensors and digital light projection technology have facilitated rapid progress in 3D optical sensing,enabling 3D surfaces of complexshaped objects to be captured with high resolution and accuracy.Nevertheless,due to the inherent synchronous pattern projection and image acquisition mechanism,the temporal resolution of conventional structured light or fringe projection profilometry(FPP)based 3D imaging methods is still limited to the native detector frame rates.In this work,we demonstrate a new 3D imaging method,termed deep-learning-enabled multiplexed FPP(DLMFPP),that allows to achieve high-resolution and high-speed 3D imaging at near-one-order of magnitude-higher 3D frame rate with conventional low-speed cameras.By encoding temporal information in one multiplexed fringe pattern,DLMFPP harnesses deep neural networks embedded with Fourier transform,phase-shifting and ensemble learning to decompose the pattern and analyze separate fringes,furnishing a high signal-to-noise ratio and a ready-to-implement solution over conventional computational imaging techniques.We demonstrate this method by measuring different types of transient scenes,including rotating fan blades and bullet fired from a toy gun,at kHz using cameras of around 100 Hz.Experiential results establish that DLMFPP allows slow-scan cameras with their known advantages in terms of cost and spatial resolution to be used for high-speed 3D imaging tasks.
基金funded by the National Science Foundation(NSF)Award 2009384.
文摘In 2019,the Event Horizon Telescope(EHT)released the first-ever image of a black hole event horizon.Astronomers are now aiming for higher angular resolutions of distant targets,like black holes,to understand more about the fundamental laws of gravity that govern our universe.To achieve this higher resolution and increased sensitivity,larger radio telescopes are needed to operate at higher frequencies and in larger quantities.Projects like the next-generation Very Large Array(ngVLA)and the Square-Kilometer Array(SKA)require building hundreds of telescopes with diameters greater than 10 ms over the next decade.This has a twofold effect.Radio telescope surfaces need to be more accurate to operate at higher frequencies,and the logistics involved in maintaining a radio telescope need to be simplified to support them properly in large quantities.Both of these problems can be solved with improved methods for surface metrology that are faster and more accurate with a higher resolution.This leads to faster and more accurate panel alignment and,therefore,a more productive observatory.In this paper,we present the use of binocular fringe projection profilometry as a solution to this problem and demonstrate it by aligning two panels on a 3-m radio telescope dish.The measurement takes only 10 min and directly delivers feedback on the tip,tilt,and piston of each panel to create the ideal reflector shape.
基金supported by the National Key Research and Development Program of China(No.2018YFB2001400)the Innovation Group Science Fund of Chongqing Natural Science Foundation(No.cstc2019jcyj-cxttX0003)。
文摘Fringe projection profilometry(FPP)has been extensively studied in the field of three-dimensional(3D)measurement.Although FPP always uses high-frequency fringes to ensure high measurement accuracy,too many patterns are projected to unwrap the phase,which affects the speed of 3D reconstruction.We propose a high-speed 3D shape measurement method using only three high-frequency inner shifting-phase patterns(70 periods),which satisfies both high precision and high measuring speed requirements.Besides,our proposed method obtains the wrapped phase and the fringe order simultaneously without any other information and constraints.The proposed method has successfully reconstructed moving objects with high speed at the camera's full frame rate(1700 frames per second).
基金Supported by National Defense Basic Scientific Research Program of China(Grant No.JCKY2021602B032)。
文摘Fringe projection profilometry(FPP)has been widely applied to non-contact three-dimensional measurement in industries owing to its high accuracy and speed.The point cloud,which is a measurement result of the FPP system,typically contains a large number of invalid points caused by the background,ambient light,shadows,and object edge regions.Research on noisy point detection and elimination has been conducted over the past two decades.However,existing invalid point removal methods are based on image intensity analysis and are only applicable to simple measurement backgrounds that are purely dark.In this paper,we propose a novel invalid point removal framework that consists of two aspects:(1)A convolutional neural network(CNN)is designed to segment the foreground from the background of different intensity conditions in FPP measurement circumstances to remove background points and the most discrete points in background regions.(2)A two-step method based on the fringe image intensity threshold and a bilateral filter is proposed to eliminate the small number of discrete points remaining after background segmentation caused by shadows and edge areas on objects.Experimental results verify that the proposed framework(1)can remove background points intelligently and accurately in different types of complex circumstances,and(2)performs excellently in discrete point detection from object regions.
文摘The Fringe Projection Profilometry(FPP)system with a single exposure time or a single projection intensity is limited by the dynamic range of the camera,which can lead to overexposure and underexposure of the image,resulting in point cloud loss or reduced accuracy.To address this issue,unlike the pixel modulation method of projectors,we utilize the characteristics of color projectors where the intensity of the three-channel LED can be controlled independently.We propose a method for separating the projector's three-channel light intensity,combined with a color camera,to achieve single exposure and multi-intensity image acquisition.Further,the crosstalk coefficient is applied to predict the three-channel reflectance of the measured object.By integrating clustering and channel mapping,we establish a pixel-level mapping model between the projector's three-channel current and the camera's three-channel image intensity,which realizes the optimal projection current prediction and the high dynamic range(HDR)image acquisition.The proposed method allows for high-precision three-dimensional(3D)data acquisition of HDR scenes with a single exposure.The effectiveness of this method has been validated through experiments with standard planes and standard steps,showing a significant reduction in mean absolute error(44.6%)compared to existing singleexposure HDR methods.Additionally,the number of images required for acquisition is significantly reduced(by 70.8%)compared to multi-exposure fusion methods.This proposed method has great potential in various FPP-related fields.
基金funded by Shenzhen Science and Technology Program(Grant JCYJ20240813112003005)by The Major Key Project of Pengcheng Laboratory(PCL2023A09).
文摘In fringe projection profilometry 3D measurement systems,the measurement of surfaces with high variability in reflectivity poses a challenge due to the limited dynamic range of cameras.The main solution involves using multiple exposures to modulate fringe intensity;however,it is inefficient.In this study,we introduce an attention-guided end-to-end phase calculation network to accelerate the multi-exposure structured light process for high dynamic range(HDR)measurements.We use attention modules to guide feature selection,enhancing relevant features and suppressing irrelevant features.Using the 12-step phase-shifting profilometry(PSP)as ground truth,our method accurately extracts the sine and cosine components of the fundamental frequency from a single pattern to retrieve the absolute phases.Tested on our metallic dataset requiring HDR imaging,our method achieves an absolute phase error of 0.084,close to that of the six-step PSP method(0.069),while using only 16.7%of the time.On the ceramic dataset,our method achieves 0.021 phase error,close to that of the four-step PSP(0.012).In quantitative measurements,our method achieves an accuracy of approximately 40μm on standard spheres and plates.Overall,our method preserves the accuracy of multi-exposure PSP methods while significantly accelerating the 3D reconstruction process.
