近三年论文 · 35 篇 (点击展开摘要,时间倒序)
Simultaneous Label-free Imaging of Nucleolar Dynamics and Subcellular Metabolic Shifts Across Tissue Contexts
The nucleolus is essential for ribosome biogenesis and for regulating cellular responses to growth and stress, and its relationship to cellular metabolic activity and functional state highlights its potential as a biomarker of cellular health. However, challenges in contrast multiplexing and high-resolution isotropic three-dimensional (3D) imaging hinder the non-invasive, simultaneous assessment of nucleolar activity and subcellular metabolic maps across different tissue contexts, especially in complex 3D environments. To fully harness the nucleolus's potential as a biomarker and diagnostic target, we present a multimodal imaging platform that combines third harmonic generation (THG) imaging with metabolic autofluorescence of NAD(P)H and FAD to study structural and metabolic nucleolar dynamics. Enabled by a high-power multimode fiber source and an axial deblurring network, we achieved ∼ 400 nm isotropic resolution in deep 3D imaging and confirmed the high accuracy of our method for label-free nucleolus identification using co-registered immunostaining and electron microscopy. To establish the biological relevance of our approach, we demonstrate that nucleolar stress leads to an unexpected depletion of NADH across cellular compartments. Furthermore, in the human endometrium-where nucleolar dynamics are central to the tissue's response to progesterone-our label-free imaging strategy delineated endometrial structures in freshly excised tissues and revealed that progesterone treatment induces distinct changes in nucleolar translocation and metabolic adaptation in organoids derived from diseased patients compared to controls. This capacity to non-invasively visualize and quantify features of the nucleolus and its local metabolic microenvironment at single-cell resolution in human tissues-and dynamically track these changes over time in patient-derived organoids-provides a powerful tool for uncovering the roles of the nucleolus in development, disease progression, and therapeutic response. Together, these findings establish our platform as a significant advance for both fundamental research and organelle-based tissue diagnostics.
Roadmap on deep learning for microscopy
Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.
Ring deconvolution microscopy: exploiting symmetry for efficient spatially varying aberration correction
The most ubiquitous form of aberration correction for microscopy is deconvolution; however, deconvolution relies on the assumption that the system's point spread function is the same across the entire field of view. This assumption is often inadequate, but space-variant deblurring techniques generally require impractical amounts of calibration and computation. We present an imaging pipeline that leverages symmetry to provide simple and fast spatially varying deblurring. Our ring deconvolution microscopy method utilizes the rotational symmetry of most microscopes and cameras, and naturally extends to sheet deconvolution in the case of lateral symmetry. We derive theory and algorithms for ring deconvolution microscopy and propose a neural network based on Seidel aberration coefficients as a fast alternative. We demonstrate improvements in speed and image quality as compared to standard deconvolution and existing spatially varying deblurring across a diverse range of microscope modalities, including miniature microscopy, multicolor fluorescence microscopy, multimode fiber micro-endoscopy and light-sheet fluorescence microscopy. Our approach enables near-isotropic, subcellular resolution in each of these applications.
High-throughput multiphoton microscopy with multimode fibers
Adaptive multimode fiber source for deep nonlinear microscopy
Multimode optical fibers (MMFs) are efficient for high-dimensional data transmission and high-power delivery by supporting multiple spatial modes, with broad applications in optical computing and bioimaging. However, inevitable random mode coupling degrades their spatiotemporal properties, despite their high spectral energy, hindering practical applications in imaging, particularly in nonlinear microscopy. Here, we report the observation of nonlinear localization in a step-index MMF at megawatt peak power levels and demonstrate its application in nonlinear microscopy. Under this nonlinear localization condition, the MMF uniquely supports the formation and propagation of this localized wave packet over long distances and flexible paths with high stability, which cannot be achieved in either single-mode fibers or condensed bulk media.
System- and sample-agnostic isotropic three-dimensional microscopy by weakly physics-informed, domain-shift-resistant axial deblurring
Three-dimensional subcellular imaging is essential for biomedical research, but the diffraction limit of optical microscopy compromises axial resolution, hindering accurate three-dimensional structural analysis. This challenge is particularly pronounced in label-free imaging of thick, heterogeneous tissues, where assumptions about data distribution (e.g. sparsity, label-specific distribution, and lateral-axial similarity) and system priors (e.g. independent and identically distributed noise and linear shift-invariant point-spread functions are often invalid. Here, we introduce SSAI-3D, a weakly physics-informed, domain-shift-resistant framework for robust isotropic three-dimensional imaging. SSAI-3D enables robust axial deblurring by generating a diverse, noise-resilient, sample-informed training dataset and sparsely fine-tuning a large pre-trained blind deblurring network. SSAI-3D is applied to label-free nonlinear imaging of living organoids, freshly excised human endometrium tissue, and mouse whisker pads, and further validated in publicly available ground-truth-paired experimental datasets of three-dimensional heterogeneous biological tissues with unknown blurring and noise across different microscopy systems. Three-dimensional imaging is crucial for biomedical research, yet microscopy faces axial resolution limitations. Here, authors introduce SSAI-3D that adapts training datasets and sparsely finetunes a network to achieve robust results across various biological samples and microscopy systems.
Adaptive Multimode Fiber Source for Label-Free Nonlinear Microscopy
We demonstrate a broadband tunable femtosecond multimode fiber source spanning 650–1350 nm, pumped by a Yb laser at 1040 nm, for multimodal label-free non-linear microscopy. Adaptive pulse optimization integrating wavefront shaping and mechanical perturbation enables high-contrast imaging with enhanced wavelength tunability and optical throughput.
Multiplexed Nonlinear Microscopy via High-Peak-Power Tunable Broadband Fiber Source
We demonstrate multiplexed nonlinear microscopy using a high-peak-power tunable broadband multimode fiber source operating in the NIR-I and NIR-II regions. Our results include labeled imaging with fluorescent beads and proteins, as well as label-free imaging with autofluorescence and harmonic generation.
Deep and Isotropic Structural and Metabolic Imaging for Nucleolar Dynamics in Living Biosystems
We develop a multimode fiber-based, label-free 3D imaging system coupled with a large-scale axial deblurring network for deep and isotropic imaging, uncovering dynamic correlations between redox states and nucleolar activities in diverse living biosystems.
Learned, uncertainty-driven adaptive acquisition for photon-efficient scanning microscopy
Scanning microscopy systems, such as confocal and multiphoton microscopy, are powerful imaging tools for probing deep into biological tissue. However, scanning systems have an inherent trade-off between acquisition time, field of view, phototoxicity, and image quality. This often results in noisy measurements when fast, large field of view, and/or gentle imaging is needed. Deep learning could be used to denoise noisy microscopy measurements, but these algorithms can be prone to hallucination, which can be disastrous for medical and scientific applications. We propose a method to simultaneously denoise and predict pixel-wise uncertainty for scanning microscopy systems, improving algorithm trustworthiness and providing statistical guarantees for deep learning predictions. Furthermore, we propose to leverage this learned, pixel-wise uncertainty to drive an adaptive acquisition technique that rescans only the most uncertain regions of a sample, saving time and reducing the total light dose to the sample. We demonstrate our method on experimental confocal and multiphoton microscopy systems, showing that our uncertainty maps can pinpoint hallucinations in the deep learned predictions. Finally, with our adaptive acquisition technique, we demonstrate up to 16× reduction in acquisition time and total light dose while successfully recovering fine features in the sample and reducing hallucinations. We are the first to demonstrate distribution-free uncertainty quantification for a denoising task with real experimental data and the first to propose adaptive acquisition based on reconstruction uncertainty.
Deep and dynamic metabolic and structural imaging in living tissues
Label-free imaging through two-photon autofluorescence of NAD(P)H allows for nondestructive, high-resolution visualization of cellular activities in living systems. However, its application to thick tissues has been restricted by its limited penetration depth within 300 μm, largely due to light scattering. Here, we demonstrate that the imaging depth for NAD(P)H can be extended to more than 700 μm in living engineered human multicellular microtissues by adopting multimode fiber-based, low repetition rate, high peak power, three-photon excitation of NAD(P)H at 1100 nm. This is achieved by having more than 0.5 megawatts peak power at the band of 1100 ± 25 nm through adaptively modulating multimodal nonlinear pulse propagation with a compact fiber shaper. Moreover, the eightfold increase in pulse energy enables faster imaging of monocyte behaviors in the living multicellular models. These results represent a substantial advance for deep and dynamic imaging of intact living biosystems. The modular design is anticipated to allow wide adoption for demanding imaging applications, including cancer research, immune responses, and tissue engineering.
High-fidelity and high-speed wavefront shaping by leveraging complex media
High-precision light manipulation is crucial for delivering information through complex media. However, existing spatial light modulation devices face a fundamental speed-fidelity tradeoff. Digital micromirror devices have emerged as a promising candidate for high-speed wavefront shaping but at the cost of compromised fidelity due to the limited control degrees of freedom. Here, we leverage the sparse-to-random transformation through complex media to overcome the dimensionality limitation of spatial light modulation devices. We demonstrate that pattern compression by sparsity-constrained wavefront optimization allows sparse and robust wavefront representations in complex media, improving the projection fidelity without sacrificing frame rate, hardware complexity, or optimization time. Our method is generalizable to different pattern types and complex media, supporting consistent performance with up to 89% and 126% improvements in projection accuracy and speckle suppression, respectively. The proposed optimization framework could enable high-fidelity high-speed wavefront shaping through different scattering media and platforms without changes to the existing holographic setups, facilitating a wide range of physics and real-world applications.
EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging
Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus restricting data throughput to maintain high SBP at limited frame rates. To address this, we introduce EventLFM, a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM), a state-of-the-art single-shot 3D wide-field imaging technique. The event camera operates on a novel asynchronous readout architecture, thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore, we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving C. elegans. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.
System- and Sample-agnostic Isotropic 3D Microscopy by Weakly Physics-informed, Domain-shift-resistant Axial Deblurring
Three-dimensional (3D) subcellular imaging is essential for biomedical research, but the diffraction limit of optical microscopy compromises axial resolution, hindering accurate 3D structural analysis. This challenge is particularly pronounced in label-free imaging of thick, heterogeneous tissues, where assumptions about data distribution (e.g. sparsity, label-specific distribution, and lateral-axial similarity) and system priors (e.g. independent and identically distributed (i.i.d.) noise and linear shift-invariant (LSI) point-spread functions (PSFs)) are often invalid. Here, we introduce SSAI-3D, a weakly physics-informed, domain-shift-resistant framework for robust isotropic 3D imaging. SSAI-3D enables robust axial deblurring by generating a PSF-flexible, noise-resilient, sample-informed training dataset and sparsely fine-tuning a large pre-trained blind deblurring network. SSAI-3D was applied to label-free nonlinear imaging of living organoids, freshly excised human endometrium tissue, and mouse whisker pads, and further validated in publicly available ground-truth-paired experimental datasets of 3D heterogeneous biological tissues with unknown blurring and noise across different microscopy systems.
Leveraging uncertainty quantification in adaptive multiphoton microscopy acquisition
Multiphoton microscopy (MPM) provides high-resolution imaging of deep tissue structures while allowing for the visualization of non-labeled biological samples. However, photon generation efficiency of intrinsic biomarkers is low and this, coupled with inherent detection inaccuracies in the photoelectric sensors, leads to an introduction of noise in acquired images. Higher dwelling times can reduce noise but increase the likelihood of photobleaching. To combat this, deep learning methods are being increasingly employed to denoise MPM images, allowing for a more efficient and less invasive process. However, machine learning models can hallucinate information, which is unacceptable for critical scientific microscopy applications. Uncertainty quantification, which has been demonstrated for image-to-image regression tasks, can provide confidence bounds for machine learning-based image reconstruction tasks, adding confidence to predictions. In this work, we discuss incorporating uncertainty quantification into an optimized denoising model to guide adaptive multiphoton microscopy image acquisition. We demonstrate that our method is capable of maintaining fine features in the denoised image, while outperforming other denoising methods by adaptively selecting to reimage the most uncertain pixels in a human endometrium tissue sample.
High-fidelity, high-speed wavefront shaping by leveraging complex media
Achieving high-precision light manipulation is crucial for delivering information through complex media with high fidelity. Digital micromirror devices (DMDs) have emerged as a promising candidate as high-speed wavefront shaping devices but at the cost of compromised fidelity, largely due to the limited degrees of freedom and the challenge of optimizing a binary amplitude mask. Here we leverage the properties of sparse-to-dense transformation in complex media and introduce a sparsity-constrained optimization framework. The proposed optimization framework could enhance existing holographic setups without changes to the hardware, and enable high-fidelity and high-speed wavefront shaping through different scattering media and platforms.
Biological discovery driven by AI-assisted label-free microscopy
Multiplexed nonlinear label-free microscopy by excitation engineering
SWIR-based microscopy has opened up windows to cellular and extracellular dynamics in deep tissues and living biological systems. New generation of laser sources, with high pulse energy, wide continuous tunable range, and a compact form, are in high demand to advance nonlinear microscopy and SWIR-based imaging to its full potential for deep-tissue imaging. This talk will discuss our recently developed approach that exploits the spatial and temporal degrees of control of nonlinear effects in step-index MMFs using a 3D-printed programmable fiber piano. By leveraging the rich spatiotemporal degrees of freedom and the high spectral brilliance in SI MMF, We have achieved broadband high-peak-power spanning 560–2200 nm, resulting from combined spectral energy reallocation (up to 166-fold) and temporal shortening (up to 4-fold) uniquely enabled by the fiber shaper. Its potential as a nonlinear imaging source is further demonstrated by applying the MMF source to multiphoton microscopy, where multi-fold signal enhancement is achieved for label-free tissue imaging with adaptive optimization.
High-speed simultaneous label-free autofluorescence-multiharmonic (hSLAM) microscopy for multicellular dynamics
Label-free nonlinear microscopy allows for high-resolution and three-dimensional imaging of live biological specimens without the need for exogenous labels. The integration of multiple modalities further enhances molecular specificity and visualization diversity for metabolic and structural mapping of heterogeneous tissue architectures. In this work, we introduce high-speed simultaneous label-free autofluorescence-multiharmonic (hSLAM) microscopy, where a high-peak-power adaptive fiber source based on multimode fiber (MMF) is employed with a nonlinear fiber piano. We will also talk about how the higher speed SLAM enables multicellular dynamics in living tissues with higher spectral flexibility and peak power, providing new possibilities for bioimaging.
Towards richer and deeper microscopy: light manipulation by leveraging complex media
Tunable broadband fiber source for multiplexed nonlinear microscopy
Multiplex imaging facilities biological studies in multicellular dynamics in living organisms due to its molecular specificity, 3D subcellular resolution, and deep tissue penetration. However, one major bottleneck is detecting and resolving multiplexed signals of weak fluorescence due to a tradeoff between signal throughput and spectral resolution. Here, we demonstrate high-speed, programmable, and broadband excitation encoding to enhance sensitivity without sacrificing signal throughput in multiplex multiphoton imaging. We utilize a 22-kHz programmable digital micromirror device to modulate the spectrum of a high-power broadband laser, achieving versatile excitation encoding schemes with a 750-nm bandwidth in the NIR regime. The proposed method will benefit applications that demand high-speed and high-content performance, including hyperspectral multiphoton microscopy and computational spectroscopy.
Spectral-temporal-spatial customization via modulating multimodal nonlinear pulse propagation
Multimode fibers (MMFs) are gaining renewed interest for nonlinear effects due to their high-dimensional spatiotemporal nonlinear dynamics and scalability for high power. High-brightness MMF sources with effective control of the nonlinear processes would offer possibilities in many areas from high-power fiber lasers, to bioimaging and chemical sensing, and to intriguing physics phenomena. Here we present a simple yet effective way of controlling nonlinear effects at high peak power levels. This is achieved by leveraging not only the spatial but also the temporal degrees of freedom during multimodal nonlinear pulse propagation in step-index MMFs, using a programmable fiber shaper that introduces time-dependent disorders. We achieve high tunability in MMF output fields, resulting in a broadband high-peak-power source. Its potential as a nonlinear imaging source is further demonstrated through widely tunable two-photon and three-photon microscopy. These demonstrations provide possibilities for technology advances in nonlinear optics, bioimaging, spectroscopy, optical computing, and material processing.
Learned, uncertainty-driven adaptive acquisition for photon-efficient scanning microscopy
Scanning microscopy systems, such as confocal and multiphoton microscopy, are powerful imaging tools for probing deep into biological tissue. However, scanning systems have an inherent trade-off between acquisition time, field of view, phototoxicity, and image quality, often resulting in noisy measurements when fast, large field of view, and/or gentle imaging is needed. Deep learning could be used to denoise noisy microscopy measurements, but these algorithms can be prone to hallucination, which can be disastrous for medical and scientific applications. We propose a method to simultaneously denoise and predict pixel-wise uncertainty for scanning microscopy systems, improving algorithm trustworthiness and providing statistical guarantees for deep learning predictions. Furthermore, we propose to leverage this learned, pixel-wise uncertainty to drive an adaptive acquisition technique that rescans only the most uncertain regions of a sample, saving time and reducing the total light dose to the sample. We demonstrate our method on experimental confocal and multiphoton microscopy systems, showing that our uncertainty maps can pinpoint hallucinations in the deep learned predictions. Finally, with our adaptive acquisition technique, we demonstrate up to 16X reduction in acquisition time and total light dose while successfully recovering fine features in the sample and reducing hallucinations. We are the first to demonstrate distribution-free uncertainty quantification for a denoising task with real experimental data and the first to propose adaptive acquisition based on reconstruction uncertainty.
EventLFM: Event Camera integrated Fourier Light Field Microscopy for Ultrafast 3D imaging
Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus restricting data throughput to maintain high SBP at limited frame rates. To address this, we introduce EventLFM, a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM), a state-of-the-art single-shot 3D wide-field imaging technique. The event camera operates on a novel asynchronous readout architecture, thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore, we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving C. elegans. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.
Spectral-temporal-spatial customization via modulating multimodal nonlinear pulse propagation
Multimode fibers (MMFs) have recently reemerged as attractive avenues for nonlinear effects due to their high-dimensional spatiotemporal nonlinear dynamics and scalability for high power. High-brightness MMF sources with effective control of the nonlinear processes would offer new possibilities for a wide range of applications from high-power fiber lasers, to bioimaging and chemical sensing, and to novel physics phenomena. Here we present a simple yet effective way of controlling nonlinear effects at high peak power levels: by leveraging not only the spatial but also the temporal degrees of freedom of the multimodal nonlinear pulse propagation in step-index MMFs using a programmable fiber shaper. This method represents the first method that enables modulation and optimization of multimodal nonlinear pulse propagation, achieving high tunability and broadband high peak power. Its potential as a nonlinear imaging source is further demonstrated by applying the MMF source to multiphoton microscopy, where widely tunable two-photon and three-photon imaging is achieved with adaptive optimization. These demonstrations highlight the effectiveness of directly modulating multimodal nonlinear pulse propagation to enhance the high-dimensional customization and optimize the high spectral brightness of MMF output. These advancements provide new possibilities for technology advances in nonlinear optics, bioimaging, spectroscopy, optical computing, and material processing.
Data from Label-Free Deep Profiling of the Tumor Microenvironment
<div>Abstract<p>Label-free nonlinear microscopy enables nonperturbative visualization of structural and metabolic contrast within living cells in their native tissue microenvironment. Here a computational pipeline was developed to provide a quantitative view of the microenvironmental architecture within cancerous tissue from label-free nonlinear microscopy images. To enable single-cell and single-extracellular vesicle (EV) analysis, individual cells, including tumor cells and various types of stromal cells, and EVs were segmented by a multiclass pixelwise segmentation neural network and subsequently analyzed for their metabolic status and molecular structure in the context of the local cellular neighborhood. By comparing cancer tissue with normal tissue, extensive tissue reorganization and formation of a patterned cell–EV neighborhood was observed in the tumor microenvironment. The proposed analytic pipeline is expected to be useful in a wide range of biomedical tasks that benefit from single-cell, single–EV, and cell-to-EV analysis.</p>Significance:<p>The proposed computational framework allows label-free microscopic analysis that quantifies the complexity and heterogeneity of the tumor microenvironment and opens possibilities for better characterization and utilization of the evolving cancer landscape.</p></div>
Supplementary File from Label-Free Deep Profiling of the Tumor Microenvironment
<p>Supplementary file</p>
Supplementary File from Label-Free Deep Profiling of the Tumor Microenvironment
<p>Supplementary file</p>
Data from Label-Free Deep Profiling of the Tumor Microenvironment
<div>Abstract<p>Label-free nonlinear microscopy enables nonperturbative visualization of structural and metabolic contrast within living cells in their native tissue microenvironment. Here a computational pipeline was developed to provide a quantitative view of the microenvironmental architecture within cancerous tissue from label-free nonlinear microscopy images. To enable single-cell and single-extracellular vesicle (EV) analysis, individual cells, including tumor cells and various types of stromal cells, and EVs were segmented by a multiclass pixelwise segmentation neural network and subsequently analyzed for their metabolic status and molecular structure in the context of the local cellular neighborhood. By comparing cancer tissue with normal tissue, extensive tissue reorganization and formation of a patterned cell–EV neighborhood was observed in the tumor microenvironment. The proposed analytic pipeline is expected to be useful in a wide range of biomedical tasks that benefit from single-cell, single–EV, and cell-to-EV analysis.</p>Significance:<p>The proposed computational framework allows label-free microscopic analysis that quantifies the complexity and heterogeneity of the tumor microenvironment and opens possibilities for better characterization and utilization of the evolving cancer landscape.</p></div>
Exploring multimode fiber sources for nonlinear microscopy and spectroscopy (Conference Presentation)
In this talk, I will present optical imaging platforms and methodologies that aim to empower label-free in vivo microscopy. Label-free in vivo microscopy promises to be a versatile tool for studying and diagnosing diseases in living animals and humans. Part of the challenge of label-free in vivo microscopy lies in the lack of simultaneous contrast, limited signal generation efficiency, and nonintuitive interpretation. This talk will cover how we attempt to address these challenges by resorting to light engineering and algorithms.
Roadmap on Deep Learning for Microscopy
Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.
High-Fidelity and High-Speed Wavefront Shaping by Leveraging Complex Media
Achieving high-precision light manipulation is crucial for delivering information through complex media with high fidelity. However, existing spatial light modulation devices face a fundamental tradeoff between speed and accuracy. Digital micromirror devices (DMDs) have emerged as a promising candidate as accessible high-speed wavefront shaping devices but at the cost of compromised fidelity, largely due to the limited control degrees of freedom and the challenge of numerically optimizing a binary amplitude mask. Here we leverage the sparse-to-random transformation through complex media to overcome the dimensionality limitation of spatial light modulation devices. We demonstrate that pattern compression in the form of sparsity-constrained wavefront optimization allows sparse and robust wavefront representations of generic patterns in the random basis provided by the complex media, and thus effectively addresses the dimensionality limitation of DMDs, which significantly improves the projection fidelity without sacrificing the full frame rate (22 kHz), hardware complexity, or optimization time (0.5 s for 1000 frames). Since the dimensionality limitation is intrinsic to spatial light modulation devices and sparse-to-random transformation to complex media, our methods can be generalized to different pattern types, complex media, and device settings, supporting consistent superior performance across different types of complex media with up to an 89% increase in projection accuracy and a 126% improvement in speckle suppression. The proposed optimization framework has the potential to enhance existing holographic setups without any change to the hardware, enable high-fidelity and high-speed wavefront shaping through different scattering media and platforms, and directly facilitate a wide range of physics and real-world applications.
Spatiotemporal Control of Nonlinear Effects in Multimode Fibers for Two-Octave High-Peak-Power Femtosecond Tunable Source
We developed a two-octave high-peak-power femtosecond tunable source by spatiotemporal control of nonlinear effects in a step-index MMF through a customer-designed fiber shaper. Its application to improve label-free imaging quality was also demonstrated.
Adaptive Fiber Source for High-Speed Label-Free Multimodal Multiphoton Microscopy
We propose a label-free multimodal multiphoton microscopy platform using a multimode fiber (MMF) source with adaptive optimization using a fiber shaper. Imaging results have shown the effectiveness of optimizing the MMF source and the potential of this platform as a versatile tool for bioimaging.
High-Fidelity and High-Speed Wavefront Shaping through Complex Media via Sparsity-Constrained Optimization
We introduce a sparsity-constrained optimization framework that accounts for the constraints on wavefront shaping and the light scattering nature in complex media to achieve high-fidelity light manipulation at a frame rate of 22 kHz. Our method demonstrates an 89% and 126% increase in projection accuracy and speckle suppression, respectively.