Multispectral imaging,which simultaneously captures the spatial and spectral information of a scene,is widely used across diverse fields,including remote sensing,biomedical imaging,and agricultural monitoring.We intro...Multispectral imaging,which simultaneously captures the spatial and spectral information of a scene,is widely used across diverse fields,including remote sensing,biomedical imaging,and agricultural monitoring.We introduce a snapshot multispectral imaging approach employing a standard monochrome image sensor with no additional spectral filters or customized components.Our system leverages the inherent chromatic aberration of wavelength-dependent defocusing as a natural source of physical encoding of multispectral information;this encoded image information is rapidly decoded via a deep learning-based multispectral Fourier imager network(mFIN).We experimentally tested our method with six illumination bands and demonstrated an overall accuracy of 98.25%for predicting the illumination channels at the input and achieved a robust multispectral image reconstruction on various test objects.This deep learning-powered framework achieves high-quality multispectral image reconstruction using snapshot image acquisition with a monochrome image sensor and could be useful for applications in biomedicine,industrial quality control,and agriculture,among others.展开更多
In an age where digitization is widespread in clinical and preclinical workflows,pathology is still predominantly practiced by microscopic evaluation of stained tissue specimens affixed on glass slides.Over the last d...In an age where digitization is widespread in clinical and preclinical workflows,pathology is still predominantly practiced by microscopic evaluation of stained tissue specimens affixed on glass slides.Over the last decade,new high throughput digital scanning microscopes have ushered in the era of digital pathology that,along with recent advances in machine vision,have opened up new possibilities for Computer-Aided-Diagnoses.Despite these advances,the high infrastructural costs related to digital pathology and the perception that the digitization process is an additional and nondirectly reimbursable step have challenged its widespread adoption.Here,we discuss how emerging virtual staining technologies and machine learning can help to disrupt the standard histopathology workflow and create new avenues for the diagnostic paradigm that will benefit patients and healthcare systems alike via digital pathology.展开更多
The immunohistochemical(IHC)staining of the human epidermal growth factor receptor 2(HER2)biomarker is widely practiced in breast tissue analysis,preclinical studies,and diagnostic decisions,guiding cancer treatment a...The immunohistochemical(IHC)staining of the human epidermal growth factor receptor 2(HER2)biomarker is widely practiced in breast tissue analysis,preclinical studies,and diagnostic decisions,guiding cancer treatment and investigation of pathogenesis.HER2 staining demands laborious tissue treatment and chemical processing performed by a histotechnologist,which typically takes one day to prepare in a laboratory,increasing analysis time and associated costs.Here,we describe a deep learning-based virtual HER2 IHC staining method using a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images,matching the standard HER2 IHC staining that is chemically performed on the same tissue sections.The efficacy of this virtual HER2 staining framework was demonstrated by quantitative analysis,in which three board-certified breast pathologists blindly graded the HER2 scores of virtually stained and immunohistochemically stained HER2 whole slide images(WSIs)to reveal that the HER2 scores determined by inspecting virtual IHC images are as accurate as their immunohistochemically stained counterparts.A second quantitative blinded study performed by the same diagnosticians further revealed that the virtually stained HER2 images exhibit a comparable staining quality in the level of nuclear detail,membrane clearness,and absence of staining artifacts with respect to their immunohistochemically stained counterparts.This virtual HER2 staining framework bypasses the costly,laborious,and time-consuming IHC staining procedures in laboratory and can be extended to other types of biomarkers to accelerate the IHC tissue staining used in life sciences and biomedical workflow.展开更多
As an optical processor,a diffractive deep neural network(D2NN)utilizes engineered diffractive surfaces designed through machine learning to perform all-optical information processing,completing its tasks at the speed...As an optical processor,a diffractive deep neural network(D2NN)utilizes engineered diffractive surfaces designed through machine learning to perform all-optical information processing,completing its tasks at the speed of light propagation through thin optical layers.With sufficient degrees of freedom,D2NNs can perform arbitrary complex-valued linear transformations using spatially coherent light.Similarly,D2NNs can also perform arbitrary linear intensity transformations with spatially incoherent illumination;however,under spatially incoherent light,these transformations are nonnegative,acting on diffraction-limited optical intensity patterns at the input field of view.Here,we expand the use of spatially incoherent D2NNs to complex-valued information processing for executing arbitrary complex-valued linear transformations using spatially incoherent light.Through simulations,we show that as the number of optimized diffractive features increases beyond a threshold dictated by the multiplication of the input and output space-bandwidth products,a spatially incoherent diffractive visual processor can approximate any complex-valued linear transformation and be used for all-optical image encryption using incoherent illumination.The findings are important for the all-optical processing of information under natural light using various forms of diffractive surface-based optical processors.展开更多
Objective and Impact Statement:Human epidermal growth factor receptor 2(HER2)is a critical protein in cancer cell growth that signifies the aggressiveness of breast cancer(BC)and helps predict its prognosis.Here,we in...Objective and Impact Statement:Human epidermal growth factor receptor 2(HER2)is a critical protein in cancer cell growth that signifies the aggressiveness of breast cancer(BC)and helps predict its prognosis.Here,we introduce a deep learning-based approach utilizing pyramid sampling for the automated classification of HER2 status in immunohistochemically(IHC)stained BC tissue images.Introduction:Accurate assessment of IHC-stained tissue slides for HER2 expression levels is essential for both treatment guidance and understanding of cancer mechanisms.Nevertheless,the traditional workflow of manual examination by board-certified pathologists encounters challenges,including inter-and intra-observer inconsistency and extended turnaround times.Methods:Our deep learning-based method analyzes morphological features at various spatial scales,efficiently managing the computational load and facilitating a detailed examination of cellular and larger-scale tissue-level details.Results:This approach addresses the tissue heterogeneity of HER2 expression by providing a comprehensive view,leading to a blind testing classification accuracy of 84.70%,on a dataset of 523 core images from tissue microarrays.Conclusion:This automated system,proving reliable as an adjunct pathology tool,has the potential to enhance diagnostic precision and evaluation speed,and might substantially impact cancer treatment planning.展开更多
In their recently published paper in Opto-Electronic Ad-vances,Pietro Ferraro and his colleagues report on a new high-throughput tomographic phase instrument that precisely quantifies intracellular lipid droplets(LDs)...In their recently published paper in Opto-Electronic Ad-vances,Pietro Ferraro and his colleagues report on a new high-throughput tomographic phase instrument that precisely quantifies intracellular lipid droplets(LDs)1.LDs are lipid storage organelles found in most cell types and play an active role in critical biological pro-cesses,including energy metabolism,membrane homeo-stasis.展开更多
Unidirectional imagers form images of input objects only in one direction,e.g.,from field-of-view(FOV)A to FOV B,while blocking the image formation in the reverse direction,from FOV B to FOV A.Here,we report unidirect...Unidirectional imagers form images of input objects only in one direction,e.g.,from field-of-view(FOV)A to FOV B,while blocking the image formation in the reverse direction,from FOV B to FOV A.Here,we report unidirectional imaging under spatially partially coherent light and demonstrate high-quality imaging only in the forward direction(A→B)with high power efficiency while distorting the image formation in the backward direction(B→A)along with low power efficiency.Our reciprocal design features a set of spatially engineered linear diffractive layers that are statistically optimized for partially coherent illumination with a given phase correlation length.Our analyses reveal that when illuminated by a partially coherent beam with a correlation length of≥∼1.5λ,whereλis the wavelength of light,diffractive unidirectional imagers achieve robust performance,exhibiting asymmetric imaging performance between the forward and backward directions—as desired.A partially coherent unidirectional imager designed with a smaller correlation length of<1.5λstill supports unidirectional image transmission but with a reduced figure of merit.These partially coherent diffractive unidirectional imagers are compact(axially spanning<75λ),polarization-independent,and compatible with various types of illumination sources,making them well-suited for applications in asymmetric visual information processing and communication.展开更多
Optical computing holds promise for high-speed,energy-efficient information processing,with diffractive optical networks emerging as a flexible platform for implementing task-specific transformations.A challenge,howev...Optical computing holds promise for high-speed,energy-efficient information processing,with diffractive optical networks emerging as a flexible platform for implementing task-specific transformations.A challenge,however,is the effective optimization and alignment of the diffractive layers,which is hindered by the difficulty of accurately modeling physical systems with their inherent hardware imperfections,noise,and misalignments.While existing in situ optimization methods offer the advantage of direct training on the physical system without explicit system modeling,they are often limited by slow convergence and unstable performance due to inefficient use of limited measurement data.Here,we introduce a model-free reinforcement learning approach utilizing Proximal Policy Optimization(PPO)for the in situ training of diffractive optical processors.PPO efficiently reuses in situ measurement data and constrains policy updates to ensure more stable and faster convergence.We validated our method through both simulations and experiments across a range of in situ learning tasks,including targeted energy focusing through a random diffuser,image generation,aberration correction,and optical image classification,demonstrating in each task better convergence and performance.Our strategy operates directly on the physical system and naturally accounts for unknown real-world imperfections,eliminating the need for prior system knowledge or modeling.By enabling faster and more accurate training under realistic experimental constraints,this in situ reinforcement learning approach could offer a scalable framework for various optical and physical systems governed by complex,feedback-driven dynamics.展开更多
We present a broadband and polarization-insensitive unidirectional imager that operates at the visible part of the spectrum,where image formation occurs in one direction,while in the opposite direction,it is blocked.T...We present a broadband and polarization-insensitive unidirectional imager that operates at the visible part of the spectrum,where image formation occurs in one direction,while in the opposite direction,it is blocked.This approach is enabled by deep learning-driven diffractive optical design with wafer-scale nano-fabrication using high-purity fused silica to ensure optical transparency and thermal stability.Our design achieves unidirectional imaging across three visible wavelengths(covering red,green,and blue parts of the spectrum),and we experimentally validated this broadband unidirectional imager by creating high-fidelity images in the forward direction and generating weak,distorted output patterns in the backward direction,in alignment with our numerical simulations.This work demonstrates wafer-scale production of diffractive optical processors,featuring 16 levels of nanoscale phase features distributed across two axially aligned diffractive layers for visible unidirectional imaging.This approach facilitates mass-scale production of~0.5 billion nanoscale phase features per wafer,supporting high-throughput manufacturing of hundreds to thousands of multi-layer diffractive processors suitable for large apertures and parallel processing of multiple tasks.Beyond broadband unidirectional imaging in the visible spectrum,this study establishes a pathway for artificial-intelligence-enabled diffractive optics with versatile applications,signaling a new era in optical device functionality with industrial-level,massively scalable fabrication.展开更多
Point spread function(PSF)engineering has been pivotal in the remarkable progress made in high-resolution imaging in the last decades.However,the diversity in PSF structures attainable through existing engineering met...Point spread function(PSF)engineering has been pivotal in the remarkable progress made in high-resolution imaging in the last decades.However,the diversity in PSF structures attainable through existing engineering methods is limited.Here,we report universal PSF engineering,demonstrating a method to synthesize an arbitrary set of spatially varying 3D PSFs between the input and output volumes of a spatially incoherent diffractive processor composed of cascaded transmissive surfaces.We rigorously analyze the PSF engineering capabilities of such diffractive processors within the diffraction limit of light and provide numerical demonstrations of unique imaging capabilities,such as snapshot 3D multispectral imaging without involving any spectral filters,axial scanning or digital reconstruction steps,which is enabled by the spatial and spectral engineering of 3D PSFs.Our framework and analysis would be important for future advancements in computational imaging,sensing,and diffractive processing of 3D optical information.展开更多
Phase recovery from intensity-only measurements forms the heart of coherent imaging techniques and holography.In this study,we demonstrate that a neural network can learn to perform phase recovery and holographic imag...Phase recovery from intensity-only measurements forms the heart of coherent imaging techniques and holography.In this study,we demonstrate that a neural network can learn to perform phase recovery and holographic image reconstruction after appropriate training.This deep learning-based approach provides an entirely new framework to conduct holographic imaging by rapidly eliminating twin-image and self-interference-related spatial artifacts.This neural network-based method is fast to compute and reconstructs phase and amplitude images of the objects using only one hologram,requiring fewer measurements in addition to being computationally faster.We validated this method by reconstructing the phase and amplitude images of various samples,including blood and Pap smears and tissue sections.These results highlight that challenging problems in imaging science can be overcome through machine learning,providing new avenues to design powerful computational imaging systems.展开更多
Deep learning has been transformative in many fields,motivating the emergence of various optical computing architectures.Diffractive optical network is a recently introduced optical computing framework that merges wav...Deep learning has been transformative in many fields,motivating the emergence of various optical computing architectures.Diffractive optical network is a recently introduced optical computing framework that merges wave optics with deep-learning methods to design optical neural networks.Diffraction-based all-optical object recognition systems,designed through this framework and fabricated by 3D printing,have been reported to recognize handwritten digits and fashion products,demonstrating all-optical inference and generalization to sub-classes of data.These previous diffractive approaches employed monochromatic coherent light as the illumination source.Here,we report a broadband diffractive optical neural network design that simultaneously processes a continuum of wavelengths generated by a temporally incoherent broadband source to all-optically perform a specific task learned using deep learning.We experimentally validated the success of this broadband diffractive neural network architecture by designing,fabricating and testing seven different multi-layer,diffractive optical systems that transform the optical wavefront generated by a broadband THz pulse to realize(1)a series of tuneable,single-passband and dual-passband spectral filters and(2)spatially controlled wavelength de-multiplexing.Merging the native or engineered dispersion of various material systems with a deep-learning-based design strategy,broadband diffractive neural networks help us engineer the light–matter interaction in 3D,diverging from intuitive and analytical design methods to create taskspecific optical components that can all-optically perform deterministic tasks or statistical inference for optical machine learning.展开更多
Recent advances in deep learning have given rise to a new paradigm of holographic image reconstruction and phase recovery techniques with real-time performance.Through data-driven approaches,these emerging techniques ...Recent advances in deep learning have given rise to a new paradigm of holographic image reconstruction and phase recovery techniques with real-time performance.Through data-driven approaches,these emerging techniques have overcome some of the challenges associated with existing holographic image reconstruction methods while also minimizing the hardware requirements of holography.These recent advances open up a myriad of new opportunities for the use of coherent imaging systems in biomedical and engineering research and related applications.展开更多
Using a deep neural network,we demonstrate a digital staining technique,which we term PhaseStain,to transform the quantitative phase images(QPI)of label-free tissue sections into images that are equivalent to the brig...Using a deep neural network,we demonstrate a digital staining technique,which we term PhaseStain,to transform the quantitative phase images(QPI)of label-free tissue sections into images that are equivalent to the brightfield microscopy images of the same samples that are histologically stained.Through pairs of image data(QPI and the corresponding brightfield images,acquired after staining),we train a generative adversarial network and demonstrate the effectiveness of this virtual-staining approach using sections of human skin,kidney,and liver tissue,matching the brightfield microscopy images of the same samples stained with Hematoxylin and Eosin,Jones’stain,and Masson’s trichrome stain,respectively.This digital-staining framework may further strengthen various uses of label-free QPI techniques in pathology applications and biomedical research in general,by eliminating the need for histological staining,reducing sample preparation related costs and saving time.Our results provide a powerful example of some of the unique opportunities created by data-driven image transformations enabled by deep learning.展开更多
Wide field-of-view(FOV)and high-resolution imaging requires microscopy modalities to have large space-bandwidth products.Lensfree on-chip microscopy decouples resolution from FOV and can achieve a space-bandwidth prod...Wide field-of-view(FOV)and high-resolution imaging requires microscopy modalities to have large space-bandwidth products.Lensfree on-chip microscopy decouples resolution from FOV and can achieve a space-bandwidth product greater than one billion under unit magnification using state-of-the-art opto-electronic sensor chips and pixel super-resolution techniques.However,using vertical illumination,the effective numerical aperture(NA)that can be achieved with an on-chip microscope is limited by a poor signal-to-noise ratio(SNR)at high spatial frequencies and imaging artifacts that arise as a result of the relatively narrow acceptance angles of the sensor’s pixels.Here,we report,for the first time,a synthetic aperture-based on-chip microscope in which the illumination angle is scanned across the surface of a dome to increase the effective NA of the reconstructed lensfree image to 1.4,achieving e.g.,,250-nm resolution at 700-nm wavelength under unit magnification.This synthetic aperture approach not only represents the largest NA achieved to date using an on-chip microscope but also enables color imaging of connected tissue samples,such as pathology slides,by achieving robust phase recovery without the need for multi-height scanning or any prior information about the sample.To validate the effectiveness of this synthetic aperture-based,partially coherent,holographic on-chip microscope,we have successfully imaged color-stained cancer tissue slides as well as unstained Papanicolaou smears across a very large FOV of 20.5 mm^(2).This compact on-chip microscope based on a synthetic aperture approach could be useful for various applications in medicine,physical sciences and engineering that demand high-resolution wide-field imaging.展开更多
A plethora of research advances have emerged in the fields of optics and photonics that benefit from harnessing the power of machine learning.Specifically,there has been a revival of interest in optical computing hard...A plethora of research advances have emerged in the fields of optics and photonics that benefit from harnessing the power of machine learning.Specifically,there has been a revival of interest in optical computing hardware due to its potential advantages for machine learning tasks in terms of parallelization,power efficiency and computation speed.Diffractive deep neural networks(D^(2)NNs)form such an optical computing framework that benefits from deep learning-based design of successive diffractive layers to all-optically process information as the input light diffracts through these passive layers.D^(2)NNs have demonstrated success in various tasks,including object classification,the spectral encoding of information,optical pulse shaping and imaging.Here,we substantially improve the inference performance of diffractive optical networks using feature engineering and ensemble learning.After independently training 1252 D^(2)NNs that were diversely engineered with a variety of passive input filters,we applied a pruning algorithm to select an optimized ensemble of D^(2)NNs that collectively improved the image classification accuracy.Through this pruning,we numerically demonstrated that ensembles of N=14 and N=30 D^(2)NNs achieve blind testing accuracies of 61.14±0.23%and 62.13±0.05%,respectively,on the classification of GFAR-10 test images,providing an inference improvennent of>16%compared to the average performance of the individual D^(2)NNs within each ensemble.These results constitute the highest inference accuracies achieved to date by any diffractive optical neural network design on the same dataset and might provide a significant leap to extend the application space of diffractive optical image classification and machine vision systems.展开更多
Large-scale linear operations are the cornerstone for performing complex computational tasks.Using optical computing to perform linear transformations offers potential advantages in terms of speed,parallelism,and scal...Large-scale linear operations are the cornerstone for performing complex computational tasks.Using optical computing to perform linear transformations offers potential advantages in terms of speed,parallelism,and scalability.Previously,the design of successive spatially engineered diffractive surfaces forming an optical network was demonstrated to perform statistical inference and compute an arbitrary complex-valued linear transformation using narrowband illumination.We report deep-learning-based design of a massively parallel broadband diffractive neural network for all-optically performing a large group of arbitrarily selected,complex-valued linear transformations between an input and output field of view,each with Ni and No pixels,respectively.This broadband diffractive processor is composed of Nw wavelength channels,each of which is uniquely assigned to a distinct target transformation;a large set of arbitrarily selected linear transformations can be individually performed through the same diffractive network at different illumination wavelengths,either simultaneously or sequentially(wavelength scanning).We demonstrate that such a broadband diffractive network,regardless of its material dispersion,can successfully approximate Nw unique complex-valued linear transforms with a negligible error when the number of diffractive neurons(N)in its design is≥2NwNiNo.We further report that the spectral multiplexing capability can be increased by increasing N;our numerical analyses confirm these conclusions for Nw>180 and indicate that it can further increase to Nw∼2000,depending on the upper bound of the approximation error.Massively parallel,wavelength-multiplexed diffractive networks will be useful for designing highthroughput intelligent machine-vision systems and hyperspectral processors that can perform statistical inference and analyze objects/scenes with unique spectral properties.展开更多
Research on optical computing has recently attracted significant attention due to the transformative advances in machine learning.Among different approaches,diffractive optical networks composed of spatially-engineere...Research on optical computing has recently attracted significant attention due to the transformative advances in machine learning.Among different approaches,diffractive optical networks composed of spatially-engineered transmissive surfaces have been demonstrated for all-optical statistical inference and performing arbitrary linear transformations using passive,free-space optical layers.Here,we introduce a polarization-multiplexed diffractive processor to all-optically perform multiple,arbitrarily-selected linear transformations through a single diffractive network trained using deep learning.In this framework,an array of pre-selected linear polarizers is positioned between trainable transmissive diffractive materials that are isotropic,and different target linear transformations(complex-valued)are uniquely assigned to different combinations of input/output polarization states.The transmission layers of this polarization-multiplexed diffractive network are trained and optimized via deep learning and error-backpropagation by using thousands of examples of the input/output fields corresponding to each one of the complex-valued linear transformations assigned to diffferent input/output polarization combinations.Our results and analysis reveal that a single diffractive network can successfully approximate and all-optically implement a group of arbitrarily-selected target transformations with a negligible error when the number of trainable diffractive features/neurons(N)approaches N_(p)N_(i)N_(o),where Ni and N_(o) represent the number of pixels at the input and output fields-of-view,respectively,and N_(p) refers to the number of unique linear transformations assigned to different input/output polarization combinations.This polarization-multiplexed all-optical diffractive processor can find various applications in optical computing and polarization-based machine vision tasks.展开更多
The precise engineering of materials and surfaces has been at the heart of some of the recent advances in optics and photonics.These advances related to the engineering of materials with new functionalities have also ...The precise engineering of materials and surfaces has been at the heart of some of the recent advances in optics and photonics.These advances related to the engineering of materials with new functionalities have also opened up exciting avenues for designing trainable surfaces that can perform computation and machine-learning tasks through light-matter interactions and diffraction.Here,we analyze the information-processing capacity of coherent optical networks formed by diffractive surfaces that are trained to perform an all-optical computational task between a given input and output field-of-view.We show that the dimensionality of the all-optical solution space covering the complex-valued transformations between the input and output fields-of-view is linearly proportional to the number of diffractive surfaces within the optical network,up to a limit that is dictated by the extent of the input and output fields-of-view.Deeper diffractive networks that are composed of larger numbers of trainable surfaces can cover a higher-dimensional subspace of the complex-valued linear transformations between a larger input field-of-view and a larger output field-of-view and exhibit depth advantages in terms of their statistical inference,learning,and generalization capabilities for different image classification tasks when compared with a single trainable diffractive surface.These analyses and conclusions are broadly applicable to various forms of diffractive surfaces,including,e.g.,plasmomc and/or dielectric-based metasurfaces and flat optics,which can be used to form all-optical processors.展开更多
Digital holographic microscopy enables the 3D reconstruction of volumetric samples from a single-snapshot hologram.However,unlike a conventional bright-field microscopy image,the quality of holographic reconstructions...Digital holographic microscopy enables the 3D reconstruction of volumetric samples from a single-snapshot hologram.However,unlike a conventional bright-field microscopy image,the quality of holographic reconstructions is compromised by interference fringes as a result of twin images and out-of-plane objects.Here,we demonstrate that cross-modality deep learning using a generative adversarial network(GAN)can endow holographic images of a sample volume with bright-field microscopy contrast,combining the volumetric imaging capability of holography with the speckle-and artifact-free image contrast of incoherent bright-field microscopy.We illustrate the performance of this“bright-field holography”method through the snapshot imaging of bioaerosols distributed in 3D,matching the artifact-free image contrast and axial sectioning performance of a high-NA bright-field microscope.This data-driven deep-learning-based imaging method bridges the contrast gap between coherent and incoherent imaging,and enables the snapshot 3D imaging of objects with bright-field contrast from a single hologram,benefiting from the wave-propagation framework of holography.展开更多
文摘Multispectral imaging,which simultaneously captures the spatial and spectral information of a scene,is widely used across diverse fields,including remote sensing,biomedical imaging,and agricultural monitoring.We introduce a snapshot multispectral imaging approach employing a standard monochrome image sensor with no additional spectral filters or customized components.Our system leverages the inherent chromatic aberration of wavelength-dependent defocusing as a natural source of physical encoding of multispectral information;this encoded image information is rapidly decoded via a deep learning-based multispectral Fourier imager network(mFIN).We experimentally tested our method with six illumination bands and demonstrated an overall accuracy of 98.25%for predicting the illumination channels at the input and achieved a robust multispectral image reconstruction on various test objects.This deep learning-powered framework achieves high-quality multispectral image reconstruction using snapshot image acquisition with a monochrome image sensor and could be useful for applications in biomedicine,industrial quality control,and agriculture,among others.
基金This study was financially supported by the NSF Biophotonics Program (USA).
文摘In an age where digitization is widespread in clinical and preclinical workflows,pathology is still predominantly practiced by microscopic evaluation of stained tissue specimens affixed on glass slides.Over the last decade,new high throughput digital scanning microscopes have ushered in the era of digital pathology that,along with recent advances in machine vision,have opened up new possibilities for Computer-Aided-Diagnoses.Despite these advances,the high infrastructural costs related to digital pathology and the perception that the digitization process is an additional and nondirectly reimbursable step have challenged its widespread adoption.Here,we discuss how emerging virtual staining technologies and machine learning can help to disrupt the standard histopathology workflow and create new avenues for the diagnostic paradigm that will benefit patients and healthcare systems alike via digital pathology.
基金support of NSF Biophotonics Program and the NIH/National Center for Advancing Translational Science UCLA CTSI Grant UL1TR001881.
文摘The immunohistochemical(IHC)staining of the human epidermal growth factor receptor 2(HER2)biomarker is widely practiced in breast tissue analysis,preclinical studies,and diagnostic decisions,guiding cancer treatment and investigation of pathogenesis.HER2 staining demands laborious tissue treatment and chemical processing performed by a histotechnologist,which typically takes one day to prepare in a laboratory,increasing analysis time and associated costs.Here,we describe a deep learning-based virtual HER2 IHC staining method using a conditional generative adversarial network that is trained to rapidly transform autofluorescence microscopic images of unlabeled/label-free breast tissue sections into bright-field equivalent microscopic images,matching the standard HER2 IHC staining that is chemically performed on the same tissue sections.The efficacy of this virtual HER2 staining framework was demonstrated by quantitative analysis,in which three board-certified breast pathologists blindly graded the HER2 scores of virtually stained and immunohistochemically stained HER2 whole slide images(WSIs)to reveal that the HER2 scores determined by inspecting virtual IHC images are as accurate as their immunohistochemically stained counterparts.A second quantitative blinded study performed by the same diagnosticians further revealed that the virtually stained HER2 images exhibit a comparable staining quality in the level of nuclear detail,membrane clearness,and absence of staining artifacts with respect to their immunohistochemically stained counterparts.This virtual HER2 staining framework bypasses the costly,laborious,and time-consuming IHC staining procedures in laboratory and can be extended to other types of biomarkers to accelerate the IHC tissue staining used in life sciences and biomedical workflow.
基金support of the U.S.Department of Energy (DOE),Office of Basic Energy Sciences,Division of Materials Sciences and Engineering under Award#DE-SC0023088.
文摘As an optical processor,a diffractive deep neural network(D2NN)utilizes engineered diffractive surfaces designed through machine learning to perform all-optical information processing,completing its tasks at the speed of light propagation through thin optical layers.With sufficient degrees of freedom,D2NNs can perform arbitrary complex-valued linear transformations using spatially coherent light.Similarly,D2NNs can also perform arbitrary linear intensity transformations with spatially incoherent illumination;however,under spatially incoherent light,these transformations are nonnegative,acting on diffraction-limited optical intensity patterns at the input field of view.Here,we expand the use of spatially incoherent D2NNs to complex-valued information processing for executing arbitrary complex-valued linear transformations using spatially incoherent light.Through simulations,we show that as the number of optimized diffractive features increases beyond a threshold dictated by the multiplication of the input and output space-bandwidth products,a spatially incoherent diffractive visual processor can approximate any complex-valued linear transformation and be used for all-optical image encryption using incoherent illumination.The findings are important for the all-optical processing of information under natural light using various forms of diffractive surface-based optical processors.
基金supported by the NSF Biophotonics Program(to A.O.)the NIH National Center for Interventional Biophotonic Technologies(P41 to A.O.).
文摘Objective and Impact Statement:Human epidermal growth factor receptor 2(HER2)is a critical protein in cancer cell growth that signifies the aggressiveness of breast cancer(BC)and helps predict its prognosis.Here,we introduce a deep learning-based approach utilizing pyramid sampling for the automated classification of HER2 status in immunohistochemically(IHC)stained BC tissue images.Introduction:Accurate assessment of IHC-stained tissue slides for HER2 expression levels is essential for both treatment guidance and understanding of cancer mechanisms.Nevertheless,the traditional workflow of manual examination by board-certified pathologists encounters challenges,including inter-and intra-observer inconsistency and extended turnaround times.Methods:Our deep learning-based method analyzes morphological features at various spatial scales,efficiently managing the computational load and facilitating a detailed examination of cellular and larger-scale tissue-level details.Results:This approach addresses the tissue heterogeneity of HER2 expression by providing a comprehensive view,leading to a blind testing classification accuracy of 84.70%,on a dataset of 523 core images from tissue microarrays.Conclusion:This automated system,proving reliable as an adjunct pathology tool,has the potential to enhance diagnostic precision and evaluation speed,and might substantially impact cancer treatment planning.
文摘In their recently published paper in Opto-Electronic Ad-vances,Pietro Ferraro and his colleagues report on a new high-throughput tomographic phase instrument that precisely quantifies intracellular lipid droplets(LDs)1.LDs are lipid storage organelles found in most cell types and play an active role in critical biological pro-cesses,including energy metabolism,membrane homeo-stasis.
文摘Unidirectional imagers form images of input objects only in one direction,e.g.,from field-of-view(FOV)A to FOV B,while blocking the image formation in the reverse direction,from FOV B to FOV A.Here,we report unidirectional imaging under spatially partially coherent light and demonstrate high-quality imaging only in the forward direction(A→B)with high power efficiency while distorting the image formation in the backward direction(B→A)along with low power efficiency.Our reciprocal design features a set of spatially engineered linear diffractive layers that are statistically optimized for partially coherent illumination with a given phase correlation length.Our analyses reveal that when illuminated by a partially coherent beam with a correlation length of≥∼1.5λ,whereλis the wavelength of light,diffractive unidirectional imagers achieve robust performance,exhibiting asymmetric imaging performance between the forward and backward directions—as desired.A partially coherent unidirectional imager designed with a smaller correlation length of<1.5λstill supports unidirectional image transmission but with a reduced figure of merit.These partially coherent diffractive unidirectional imagers are compact(axially spanning<75λ),polarization-independent,and compatible with various types of illumination sources,making them well-suited for applications in asymmetric visual information processing and communication.
文摘Optical computing holds promise for high-speed,energy-efficient information processing,with diffractive optical networks emerging as a flexible platform for implementing task-specific transformations.A challenge,however,is the effective optimization and alignment of the diffractive layers,which is hindered by the difficulty of accurately modeling physical systems with their inherent hardware imperfections,noise,and misalignments.While existing in situ optimization methods offer the advantage of direct training on the physical system without explicit system modeling,they are often limited by slow convergence and unstable performance due to inefficient use of limited measurement data.Here,we introduce a model-free reinforcement learning approach utilizing Proximal Policy Optimization(PPO)for the in situ training of diffractive optical processors.PPO efficiently reuses in situ measurement data and constrains policy updates to ensure more stable and faster convergence.We validated our method through both simulations and experiments across a range of in situ learning tasks,including targeted energy focusing through a random diffuser,image generation,aberration correction,and optical image classification,demonstrating in each task better convergence and performance.Our strategy operates directly on the physical system and naturally accounts for unknown real-world imperfections,eliminating the need for prior system knowledge or modeling.By enabling faster and more accurate training under realistic experimental constraints,this in situ reinforcement learning approach could offer a scalable framework for various optical and physical systems governed by complex,feedback-driven dynamics.
基金Ozcan Lab at UCLA acknowledges the U.S.Department of Energy(DOE),Office of Basic Energy Sciences,Division of Materials Sciences and Engineering under award no.DE-SC0023088.
文摘We present a broadband and polarization-insensitive unidirectional imager that operates at the visible part of the spectrum,where image formation occurs in one direction,while in the opposite direction,it is blocked.This approach is enabled by deep learning-driven diffractive optical design with wafer-scale nano-fabrication using high-purity fused silica to ensure optical transparency and thermal stability.Our design achieves unidirectional imaging across three visible wavelengths(covering red,green,and blue parts of the spectrum),and we experimentally validated this broadband unidirectional imager by creating high-fidelity images in the forward direction and generating weak,distorted output patterns in the backward direction,in alignment with our numerical simulations.This work demonstrates wafer-scale production of diffractive optical processors,featuring 16 levels of nanoscale phase features distributed across two axially aligned diffractive layers for visible unidirectional imaging.This approach facilitates mass-scale production of~0.5 billion nanoscale phase features per wafer,supporting high-throughput manufacturing of hundreds to thousands of multi-layer diffractive processors suitable for large apertures and parallel processing of multiple tasks.Beyond broadband unidirectional imaging in the visible spectrum,this study establishes a pathway for artificial-intelligence-enabled diffractive optics with versatile applications,signaling a new era in optical device functionality with industrial-level,massively scalable fabrication.
文摘Point spread function(PSF)engineering has been pivotal in the remarkable progress made in high-resolution imaging in the last decades.However,the diversity in PSF structures attainable through existing engineering methods is limited.Here,we report universal PSF engineering,demonstrating a method to synthesize an arbitrary set of spatially varying 3D PSFs between the input and output volumes of a spatially incoherent diffractive processor composed of cascaded transmissive surfaces.We rigorously analyze the PSF engineering capabilities of such diffractive processors within the diffraction limit of light and provide numerical demonstrations of unique imaging capabilities,such as snapshot 3D multispectral imaging without involving any spectral filters,axial scanning or digital reconstruction steps,which is enabled by the spatial and spectral engineering of 3D PSFs.Our framework and analysis would be important for future advancements in computational imaging,sensing,and diffractive processing of 3D optical information.
文摘Phase recovery from intensity-only measurements forms the heart of coherent imaging techniques and holography.In this study,we demonstrate that a neural network can learn to perform phase recovery and holographic image reconstruction after appropriate training.This deep learning-based approach provides an entirely new framework to conduct holographic imaging by rapidly eliminating twin-image and self-interference-related spatial artifacts.This neural network-based method is fast to compute and reconstructs phase and amplitude images of the objects using only one hologram,requiring fewer measurements in addition to being computationally faster.We validated this method by reconstructing the phase and amplitude images of various samples,including blood and Pap smears and tissue sections.These results highlight that challenging problems in imaging science can be overcome through machine learning,providing new avenues to design powerful computational imaging systems.
文摘Deep learning has been transformative in many fields,motivating the emergence of various optical computing architectures.Diffractive optical network is a recently introduced optical computing framework that merges wave optics with deep-learning methods to design optical neural networks.Diffraction-based all-optical object recognition systems,designed through this framework and fabricated by 3D printing,have been reported to recognize handwritten digits and fashion products,demonstrating all-optical inference and generalization to sub-classes of data.These previous diffractive approaches employed monochromatic coherent light as the illumination source.Here,we report a broadband diffractive optical neural network design that simultaneously processes a continuum of wavelengths generated by a temporally incoherent broadband source to all-optically perform a specific task learned using deep learning.We experimentally validated the success of this broadband diffractive neural network architecture by designing,fabricating and testing seven different multi-layer,diffractive optical systems that transform the optical wavefront generated by a broadband THz pulse to realize(1)a series of tuneable,single-passband and dual-passband spectral filters and(2)spatially controlled wavelength de-multiplexing.Merging the native or engineered dispersion of various material systems with a deep-learning-based design strategy,broadband diffractive neural networks help us engineer the light–matter interaction in 3D,diverging from intuitive and analytical design methods to create taskspecific optical components that can all-optically perform deterministic tasks or statistical inference for optical machine learning.
文摘Recent advances in deep learning have given rise to a new paradigm of holographic image reconstruction and phase recovery techniques with real-time performance.Through data-driven approaches,these emerging techniques have overcome some of the challenges associated with existing holographic image reconstruction methods while also minimizing the hardware requirements of holography.These recent advances open up a myriad of new opportunities for the use of coherent imaging systems in biomedical and engineering research and related applications.
基金The Ozcan Research Group at UCLA acknowledges the support of NSF Engineering Research Center(ERC,PATHS-UP)the Army Research Office(ARO,W911NF-13-1-0419 and W911NF-13-1-0197)+8 种基金the ARO Life Sciences Divisionthe National Science Foundation(NSF)CBET Division Biophotonics Programthe NSF Emerging Frontiers in Research and Innovation(EFRI)Awardthe NSF INSPIRE Award,NSF Partnerships for Innovation:Building Innovation Capacity(PFI:BIC)Programthe National Institutes of Health(NIH,R21EB023115)the Howard Hughes Medical Institute(HHMI)Vodafone Americas Foundationthe Mary Kay FoundationSteven&Alexandra Cohen Foundation.
文摘Using a deep neural network,we demonstrate a digital staining technique,which we term PhaseStain,to transform the quantitative phase images(QPI)of label-free tissue sections into images that are equivalent to the brightfield microscopy images of the same samples that are histologically stained.Through pairs of image data(QPI and the corresponding brightfield images,acquired after staining),we train a generative adversarial network and demonstrate the effectiveness of this virtual-staining approach using sections of human skin,kidney,and liver tissue,matching the brightfield microscopy images of the same samples stained with Hematoxylin and Eosin,Jones’stain,and Masson’s trichrome stain,respectively.This digital-staining framework may further strengthen various uses of label-free QPI techniques in pathology applications and biomedical research in general,by eliminating the need for histological staining,reducing sample preparation related costs and saving time.Our results provide a powerful example of some of the unique opportunities created by data-driven image transformations enabled by deep learning.
基金The Ozcan Research Group at UCLA gratefully acknowledges the support of the Presidential Early Career Award for Scientists and Engineers(PECASE),the Army Research Office(AROW911NF-13-1-0419 and W911NF-13-1-0197)+2 种基金the ARO Life Sciences Division,the ARO Young Investigator Award,the National Science Foundation(NSF)CAREER Award,the NSF CBET Division Biophotonics Program,the NSF Emerging Frontiers in Research and Innovation(EFRI)Award,the NSF EAGER Award,Office of Naval Research(ONR),the Howard Hughes Medical Institute(HHMI)the National Institutes of Health(NIH)Director’s New Innovator Award DP2OD006427 from the Office of the Director,National Institutes of HealthThis work is based on research performed in a laboratory renovated by the National Science Foundation under Grant No.0963183,which is an award funded under the American Recovery and Reinvestment Act of 2009(ARRA).
文摘Wide field-of-view(FOV)and high-resolution imaging requires microscopy modalities to have large space-bandwidth products.Lensfree on-chip microscopy decouples resolution from FOV and can achieve a space-bandwidth product greater than one billion under unit magnification using state-of-the-art opto-electronic sensor chips and pixel super-resolution techniques.However,using vertical illumination,the effective numerical aperture(NA)that can be achieved with an on-chip microscope is limited by a poor signal-to-noise ratio(SNR)at high spatial frequencies and imaging artifacts that arise as a result of the relatively narrow acceptance angles of the sensor’s pixels.Here,we report,for the first time,a synthetic aperture-based on-chip microscope in which the illumination angle is scanned across the surface of a dome to increase the effective NA of the reconstructed lensfree image to 1.4,achieving e.g.,,250-nm resolution at 700-nm wavelength under unit magnification.This synthetic aperture approach not only represents the largest NA achieved to date using an on-chip microscope but also enables color imaging of connected tissue samples,such as pathology slides,by achieving robust phase recovery without the need for multi-height scanning or any prior information about the sample.To validate the effectiveness of this synthetic aperture-based,partially coherent,holographic on-chip microscope,we have successfully imaged color-stained cancer tissue slides as well as unstained Papanicolaou smears across a very large FOV of 20.5 mm^(2).This compact on-chip microscope based on a synthetic aperture approach could be useful for various applications in medicine,physical sciences and engineering that demand high-resolution wide-field imaging.
基金The Ozcan Research Group at UCLA acknowledges the support of Fujikura(Japan).
文摘A plethora of research advances have emerged in the fields of optics and photonics that benefit from harnessing the power of machine learning.Specifically,there has been a revival of interest in optical computing hardware due to its potential advantages for machine learning tasks in terms of parallelization,power efficiency and computation speed.Diffractive deep neural networks(D^(2)NNs)form such an optical computing framework that benefits from deep learning-based design of successive diffractive layers to all-optically process information as the input light diffracts through these passive layers.D^(2)NNs have demonstrated success in various tasks,including object classification,the spectral encoding of information,optical pulse shaping and imaging.Here,we substantially improve the inference performance of diffractive optical networks using feature engineering and ensemble learning.After independently training 1252 D^(2)NNs that were diversely engineered with a variety of passive input filters,we applied a pruning algorithm to select an optimized ensemble of D^(2)NNs that collectively improved the image classification accuracy.Through this pruning,we numerically demonstrated that ensembles of N=14 and N=30 D^(2)NNs achieve blind testing accuracies of 61.14±0.23%and 62.13±0.05%,respectively,on the classification of GFAR-10 test images,providing an inference improvennent of>16%compared to the average performance of the individual D^(2)NNs within each ensemble.These results constitute the highest inference accuracies achieved to date by any diffractive optical neural network design on the same dataset and might provide a significant leap to extend the application space of diffractive optical image classification and machine vision systems.
基金the US Air Force Office of Scientific Research funding(Grant No.FA9550-21-1-0324)。
文摘Large-scale linear operations are the cornerstone for performing complex computational tasks.Using optical computing to perform linear transformations offers potential advantages in terms of speed,parallelism,and scalability.Previously,the design of successive spatially engineered diffractive surfaces forming an optical network was demonstrated to perform statistical inference and compute an arbitrary complex-valued linear transformation using narrowband illumination.We report deep-learning-based design of a massively parallel broadband diffractive neural network for all-optically performing a large group of arbitrarily selected,complex-valued linear transformations between an input and output field of view,each with Ni and No pixels,respectively.This broadband diffractive processor is composed of Nw wavelength channels,each of which is uniquely assigned to a distinct target transformation;a large set of arbitrarily selected linear transformations can be individually performed through the same diffractive network at different illumination wavelengths,either simultaneously or sequentially(wavelength scanning).We demonstrate that such a broadband diffractive network,regardless of its material dispersion,can successfully approximate Nw unique complex-valued linear transforms with a negligible error when the number of diffractive neurons(N)in its design is≥2NwNiNo.We further report that the spectral multiplexing capability can be increased by increasing N;our numerical analyses confirm these conclusions for Nw>180 and indicate that it can further increase to Nw∼2000,depending on the upper bound of the approximation error.Massively parallel,wavelength-multiplexed diffractive networks will be useful for designing highthroughput intelligent machine-vision systems and hyperspectral processors that can perform statistical inference and analyze objects/scenes with unique spectral properties.
文摘Research on optical computing has recently attracted significant attention due to the transformative advances in machine learning.Among different approaches,diffractive optical networks composed of spatially-engineered transmissive surfaces have been demonstrated for all-optical statistical inference and performing arbitrary linear transformations using passive,free-space optical layers.Here,we introduce a polarization-multiplexed diffractive processor to all-optically perform multiple,arbitrarily-selected linear transformations through a single diffractive network trained using deep learning.In this framework,an array of pre-selected linear polarizers is positioned between trainable transmissive diffractive materials that are isotropic,and different target linear transformations(complex-valued)are uniquely assigned to different combinations of input/output polarization states.The transmission layers of this polarization-multiplexed diffractive network are trained and optimized via deep learning and error-backpropagation by using thousands of examples of the input/output fields corresponding to each one of the complex-valued linear transformations assigned to diffferent input/output polarization combinations.Our results and analysis reveal that a single diffractive network can successfully approximate and all-optically implement a group of arbitrarily-selected target transformations with a negligible error when the number of trainable diffractive features/neurons(N)approaches N_(p)N_(i)N_(o),where Ni and N_(o) represent the number of pixels at the input and output fields-of-view,respectively,and N_(p) refers to the number of unique linear transformations assigned to different input/output polarization combinations.This polarization-multiplexed all-optical diffractive processor can find various applications in optical computing and polarization-based machine vision tasks.
基金The Ozcan Lab at UCLA acknowledges the support of Fujikura(Japan).O.Ksupport of the Fulbright Commission of Turkey。
文摘The precise engineering of materials and surfaces has been at the heart of some of the recent advances in optics and photonics.These advances related to the engineering of materials with new functionalities have also opened up exciting avenues for designing trainable surfaces that can perform computation and machine-learning tasks through light-matter interactions and diffraction.Here,we analyze the information-processing capacity of coherent optical networks formed by diffractive surfaces that are trained to perform an all-optical computational task between a given input and output field-of-view.We show that the dimensionality of the all-optical solution space covering the complex-valued transformations between the input and output fields-of-view is linearly proportional to the number of diffractive surfaces within the optical network,up to a limit that is dictated by the extent of the input and output fields-of-view.Deeper diffractive networks that are composed of larger numbers of trainable surfaces can cover a higher-dimensional subspace of the complex-valued linear transformations between a larger input field-of-view and a larger output field-of-view and exhibit depth advantages in terms of their statistical inference,learning,and generalization capabilities for different image classification tasks when compared with a single trainable diffractive surface.These analyses and conclusions are broadly applicable to various forms of diffractive surfaces,including,e.g.,plasmomc and/or dielectric-based metasurfaces and flat optics,which can be used to form all-optical processors.
基金The Ozcan Group at UCLA acknowledges the support of the Koç Group,the National Science Foundation(PATHS-UP ERC)the Howard Hughes Medical Institute.Y.W.also acknowledges the support of the SPIE John Kiel Scholarship.
文摘Digital holographic microscopy enables the 3D reconstruction of volumetric samples from a single-snapshot hologram.However,unlike a conventional bright-field microscopy image,the quality of holographic reconstructions is compromised by interference fringes as a result of twin images and out-of-plane objects.Here,we demonstrate that cross-modality deep learning using a generative adversarial network(GAN)can endow holographic images of a sample volume with bright-field microscopy contrast,combining the volumetric imaging capability of holography with the speckle-and artifact-free image contrast of incoherent bright-field microscopy.We illustrate the performance of this“bright-field holography”method through the snapshot imaging of bioaerosols distributed in 3D,matching the artifact-free image contrast and axial sectioning performance of a high-NA bright-field microscope.This data-driven deep-learning-based imaging method bridges the contrast gap between coherent and incoherent imaging,and enables the snapshot 3D imaging of objects with bright-field contrast from a single hologram,benefiting from the wave-propagation framework of holography.
