IP Library › Granted Patent US 12,327,362
Granted Patent B2
US 12,327,362 · App. 18/543,168 · Granted Jun 10, 2025

Method and system for digital staining of label-free fluorescence images using deep learning

Inventors: Aydogan Ozcan (Los Angeles, CA); Yair Rivenson (Los Angeles, CA); Hongda Wang (Los Angeles, CA); Zhensong Wei (Los Angeles, CA)
Assignee: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
G06T7/11G06F18/2155G06N3/08G06V10/764G06V10/82G16H30/20G16H30/40G16H70/60
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,327,362
App. No.
18/543,168
Granted
Jun 10, 2025
Kind
B2
Abstract

A deep learning-based digital staining method and system are disclosed that enables the creation of digitally/virtually-stained microscopic images from label or stain-free samples based on autofluorescence images acquired using a fluorescent microscope. The system and method have particular applicability for the creation of digitally/virtually-stained whole slide images (WSIs) of unlabeled/unstained tissue samples that are analyzes by a histopathologist. The methods bypass the standard histochemical staining process, saving time and cost. This method is based on deep learning, and uses, in one embodiment, a convolutional neural network trained using a generative adversarial network model to transform fluorescence images of an unlabeled sample into an image that is equivalent to the brightfield image of the chemically stained-version of the same sample. This label-free digital staining method eliminates cumbersome and costly histochemical staining procedures and significantly simplifies tissue preparation in pathology and histology fields.

Claims (28)

1. A method of generating a digitally stained microscopic image comprising:

providing a neural network using one or more processors of a computing device, wherein the neural network is trained with a plurality of chemically stained images or image patches matched with corresponding label-free images or image patches of training samples;

obtaining one or more label-free images of a label-free test sample using a fluorescence microscope and one or more excitation light sources, wherein the detected light is emitted from at least one endogenous fluorophore or at least one endogenous emitter of the label-free test sample, and wherein each of the one or more label-free images comprises a single image per each of one or more channels;

inputting the one or more label-free images of the label-free test sample to the neural network; and

outputting the digitally stained microscopic image of the label-free test sample via the neural network.

2. The method of claim 1 , wherein the neural network comprises a plurality of neural networks.

3. The method of claim 1 , wherein the neural network is trained using a Generative Adversarial Network (GAN) model.

4. The method of claim 1 , wherein the deep neural network is trained using a generator network configured to learn statistical transformation between the matched label-free and chemically stained images or image patches of the same training sample and a network configured to discriminate between a ground truth chemically stained image of the training sample and the outputted digitally stained microscopic image of the training sample.

5. The method of claim 1 , wherein the label-free test sample comprises animal tissue, plant tissue, cells, pathogens, or biological fluid smears.

6. The method of claim 1 , wherein the neural network outputs a digitally stained microscopic image in less than one second of inputting the one or more label-free image(s).

7. The method of claim 1 , wherein the label-free test sample comprises a non-fixed tissue sample.

8. The method of claim 1 , wherein the label-free test sample comprises a fixed tissue sample.

9. The method of claim 8 , wherein the fixed tissue sample is embedded in paraffin.

10. The method of claim 1 , wherein the label-free test sample comprises a frozen tissue sample.

11. The method of claim 1 , wherein the label-free test sample comprises a fresh tissue sample.

12. The method of claim 1 , wherein the excitation light source emits ultra-violet or near ultra-violet light.

13. The method of claim 1 , wherein the one or more label-free images are obtained using one or more spectral filters of a filter set.

14. The method of claim 13 , wherein a plurality of spectral filters is used to capture a plurality of label-free images.

15. The method of claim 14 , wherein the plurality of images is obtained by multiple excitation light sources emitting light at different wavelengths or wavelength bands.

16. The method of claim 1 , wherein the one or more label-free images are subject to one or more image pre-processing operations prior to being input to the neural network.

17. The method of claim 16 , wherein the one or more image pre-processing operations comprises contrast enhancement, contrast reversal, image filtering, or combinations thereof.

18. The method of claim 1 , wherein the neural network is trained using one or more GPUs or ASICs.

19. The method of claim 1 , wherein the neural network is executed using one or more GPUs or ASICs.

20. The method of claim 1 , wherein obtaining the one or more label-free images of the label-free test sample comprises obtaining at least two fluorescence images of the label-free test sample.

21. The method of claim 20 , wherein the at least two fluorescence images are obtained using different wavelengths.

22. The method of claim 20 , wherein the at least two fluorescence images are obtained using different resolutions.

23. The method of claim 1 , wherein the neural network comprises a convolutional neural network.

24. The method of claim 1 , wherein detected light is emitted from one or both of the at least one endogenous fluorophore and the at least one endogenous emitter of frequency shifted light of the label-free test sample.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2023
From: OZCAN, AYDOGAN; RIVENSON, YAIR; WANG, HONGDA; WEI, ZHENSONG
To: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
Reel/Frame 065896/0591 →
Continuity (3)
Continuation 17041447
Provisional Application 62651005 · Mar 30, 2018
Related Publication 20240135544A1 · Apr 25, 2024
References Cited (116)
US 9971966B2 · Nelson et al. · 2018 [cited by applicant]
US 10824847B2 · Wu · 2020 [cited by examiner]
US 11354804B1 · Behrooz · 2022 [cited by examiner]
US 11403735B2 · Gao · 2022 [cited by examiner]
US 11893739B2 · Ozcan et al. · 2024 [cited by applicant]
US 12106552B2 · Ozcan et al. · 2024 [cited by applicant]
US 20040093166A1 · Kil · 2004 [cited by applicant]
US 20110206262A1 · Sammak · 2011 [cited by applicant]
US 20120083678A1 · Drauch et al. · 2012 [cited by applicant]
US 20130044933A1 · Kenny · 2013 [cited by applicant]
US 20130077875A1 · Levenson et al. · 2013 [cited by applicant]
US 20130144714A1 · Yuan et al. · 2013 [cited by applicant]
US 20130317369A1 · Bryant-Greenwood · 2013 [cited by examiner]
US 20140221813A1 · Bryant-Greenwood · 2014 [cited by examiner]
US 20140270457A1 · Bhargava · 2014 [cited by examiner]
US 20150119722A1 · Kaneko · 2015 [cited by examiner]
US 20150226743A1 · Weiss · 2015 [cited by examiner]
US 20160077007A1 · Demos et al. · 2016 [cited by applicant]
US 20160196648A1 · Madabhushi · 2016 [cited by examiner]
US 20170052106A1 · Hennig et al. · 2017 [cited by applicant]
US 20170191937A1 · Levenson · 2017 [cited by examiner]
US 20170249548A1 · Nelson et al. · 2017 [cited by applicant]
US 20170343477A1 · Santori · 2017 [cited by examiner]
US 20170343784A1 · Wu · 2017 [cited by examiner]
US 20180012356A1 · Madabhushi · 2018 [cited by examiner]
US 20180028079A1 · Gurevich et al. · 2018 [cited by applicant]
US 20180064409A1 · Schmidt-Richberg et al. · 2018 [cited by applicant]
US 20180239951A1 · El-Zehiry et al. · 2018 [cited by applicant]
US 20180286038A1 · Jalali · 2018 [cited by examiner]
US 20190188446A1 · Wu · 2019 [cited by examiner]
US 20190251330A1 · Cotte · 2019 [cited by applicant]
US 20190266486A1 · Yamada · 2019 [cited by applicant]
US 20190333199A1 · Ozcan · 2019 [cited by examiner]
US 20190384047A1 · Johnson et al. · 2019 [cited by applicant]
US 20190392580A1 · Kapil et al. · 2019 [cited by applicant]
US 20200066407A1 · Stumpe et al. · 2020 [cited by applicant]
US 20200106932A1 · Chou · 2020 [cited by applicant]
US 20200372635A1 · Veidman et al. · 2020 [cited by applicant]
US 20200394825A1 · Stumpe · 2020 [cited by applicant]
US 20210264214A1 · Ozcan et al. · 2021 [cited by applicant]
US 20230030424A1 · Ozcan · 2023 [cited by examiner]
CN 1556379A · 2004 [cited by applicant]
CN 102575990A · 2012 [cited by applicant]
CN 107123137A · 2017 [cited by applicant]
EP 2921990A2 · 2015 [cited by applicant]
WO WO2013187148 · 2013 [cited by applicant]
WO WO2013187148A1 · 2013 [cited by applicant]
WO WO2017053592 · 2017 [cited by applicant]
WO WO2017196885 · 2017 [cited by applicant]
WO WO2018165590 · 2018 [cited by applicant]
WO WO2019118544A1 · 2019 [cited by applicant]
WO WO2019154987A1 · 2019 [cited by applicant]
WO WO2019191697A1 · 2019 [cited by applicant]
WO WO2020018154 · 2020 [cited by applicant]
Bayramoglu et al, Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using conditional Generative Adversarial Networks, Aug. 2017, IEEE, 2017 IEEE Conference on Computer Vision Workshops, pp. 64-71 (Yea… [cited by examiner]
PCT Notification of Transmittal of the International Search Report and Written Opinion for PCT/US2019/021471, mailed Jun. 19, 2019, 7 pages. [cited by applicant]
Bayramoglu et al., Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). Retrieve… [cited by applicant]
Rivenson et al., Deep Learning-based virtual histology staining using auto-fluorescence of label-free tissue, Nature Biomedical Engineering (2019). Retrieved on May 24, 2019.Retrieved from: <URL:https://arxiv.org/abs/18… [cited by applicant]
Croce et al., Autofluorescence spectroscopy and imaging: a tool for biomedical research and diagnosis, European Journal of Histochemistry 2014; vol. 58:2461. [cited by applicant]
Fereidouni et al., Microscopy with ultraviolet surface excitation for rapid slide-free histology, Nat Biomed Eng. 2017; 1:957-966. [cited by applicant]
Giacomelli et al., (2016) Virtual Hematoxylin and Eosin Transillumination Microscopy Using Epi-Fluorescence Imaging. PLoS ONE 11(8):e0159337. Doi:10.1371/journal.pone.0159337. [cited by applicant]
Glaser et al., Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens, Nat Biomed Eng. Jul. 2017; 1(7): doi:10.1038/s41551-017-0084. [cited by applicant]
Goodfellow et al., Generative Adversarial Nets, http://www.github.com/goodfeli/adversarial, Jun. 10, 2014. [cited by applicant]
Hamel et al., Transfer Learning in MIR: Sharing Learned Latent Representations for Music Audio Classification and Similarity, 2013 International Society for Music Information Retrieval. [cited by applicant]
He et al., Deep Residual Learning for Image Recognition, arXiv:1512.03385v1 [cs.CV] Dec. 10, 2015. [cited by applicant]
He et al., Identity Mappings in Deep Residual Networks, arXiv:1603.05027v3 [cs.CV] Jul. 25, 2016. [cited by applicant]
Jamme et al., DUV Autofluorescence Microscopy for Cell Biology and Tissue Histology Biology of the Cell Biology of the Cell, Biology of the Cell, Wiley, 2013, 105 (7), pp. 277-288. 10.1111/boc.201200075. hal-01485578. [cited by applicant]
Ji et al., Rapid, label-free detection of brain tumors with stimulated Raman scattering microscopy, Sci Transl Med. Sep. 4, 2013; 5(201): 201ra119. doi:10.1126/scitranslmed.3005954. [cited by applicant]
Kingma et al., ADAM: A Method for Stochastic Optimization, arXiv:1412.6980v9 [cs.LG] Jan. 30, 2017. [cited by applicant]
Liu et al., Detecting Cancer Metastases on Gigapixel Pathology Images, arXiv:1703.02442v2 [cs.CV] Mar. 8, 2017. [cited by applicant]
Lowe, Distinctive Image Features from Scale-Invariant Keypoints, Accepted for publication in the International Journal of Computer Vision, 2004. [cited by applicant]
Lu et al., Label-free DNA imaging in vivo with stimulated Raman scattering microscopy, 11624-11629, PNAS, Sep. 15, 2015, vol. 112, No. 37, www.pnas.org/cgi/doi/10.1073/pnas.1515121112. [cited by applicant]
Monici, Cell and tissue autofluorescence research and diagnostic application, Biotechnology Annual Review, vol. 11, ISSN: 1387-2656, DOI: 10.1016/S1387-2656(05)11007-2. [cited by applicant]
Rivenson et al., Phase recovery and holographic image reconstruction using deep learning in neural networks, Light: Science & Applications (2018) 7, 17141; doi:10.1038/lsa.2017.141. [cited by applicant]
Rivenson et al., Deep learning enhanced mobile-phone microscopy, (2017). [cited by applicant]
Rivenson et al., Deep Learning Microscopy, (2017). [cited by applicant]
Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:1505.04597v1 [cs.CV] May 18, 2015; http://lmb.informatik.uni-freiburg.de/. [cited by applicant]
Tao et al., Assessment of breast pathologies using nonlinear microscopy, 15304-15309, PNAS, Oct. 28, 2014, vol. 111, No. 43, www.pnas.org/cgi/doi/10.1073/pnas.1416955111. [cited by applicant]
Torr et al., MLESAC: A New Robust Estimator with Application to Estimating Image Geometry, Computer Vision and Image Understanding 78, 138-156 (2000), doi:10.1006/cviu.1999.0832, available online at http://www.idealibra… [cited by applicant]
Tu et al., Stain-free histopathology by programmable supercontinuum pulses, Nat Photonics. Aug. 2016 ; 10(8):534-540. doi:10.1038/nphoton.2016.94. [cited by applicant]
Vacoc et al., Three-dimensional microscopy of the tumor microenvironment in vivo using optical frequency domain imaging, Nat Med. Oct. 2009 ; 15(10): 1219-1223. doi:10.1038/nm. 1971. [cited by applicant]
Witte et al., Label-free live brain imaging and targeted patching with third-harmonic generation microscopy, 5970-5975, PNAS, Apr. 12, 2011, vol. 108, No. 15, www.pnas.org/cgi/doi/10.1073/pnas.1018743108. [cited by applicant]
Wu et al., Extended depth of field in holographic image reconstruction using deep learning based auto-focusing and phase recovery, arXiv1803.08138 Mar. 2018. [cited by applicant]
PCT Notification of Transmittal of the International Search Report and Written Opinion for PCT/US2019/025014, mailed Jun. 11, 2019, 9 pages. [cited by applicant]
PCT International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) for PCT/US2019/025020, Applicant: The Regents of the University of California, Form PCT/IB/326 and 373, dated Oct. 15, 2… [cited by applicant]
The extended European search report dated May 7, 2021 for European Patent Application No. European patent application 19776693.4 (8 pages). [cited by applicant]
Communication pursuant to Rules 70(2) and 70a(2) EPC dated May 26, 2021 for European Patent Application No. European patent application 19776693.4 (1 page). [cited by applicant]
Response to the extended European search report dated Dec. 2, 2021 for European Patent Application No. European patent application 19776693.4 (16 pages). [cited by applicant]
Neslihan Bayramoglu et al., Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW),… [cited by applicant]
Erik A. Burlingame et al., Shift: speedy histopathological-to-immunofluorescent translation of whole slide images using conditional generative adversarial networks, Progress in Biomedical Optics and Imaging, SPIE—Intern… [cited by applicant]
Liang Han et al., Transferring Microscopy Image Modalities with Conditional Generative Adversarial Networks, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jul. 21, 2017 (Jul. 21, 201… [cited by applicant]
First Examination Report dated Jul. 14, 2022 for Indian Patent Application No. 202037045753, (7 pages). [cited by applicant]
Response to the First Examination Report dated Oct. 13, 2022 for Indian Patent Application No. 202037045753, (25 pages). [cited by applicant]
Notice of Reasons for Refusal dated Jan. 18, 2023 for Japanese Patent Application No. 2020-552396, (8 pages). [cited by applicant]
Neslihan Bayramoglu et al., Towards Virtual H&E Staining of Hyperspectral Lung Histology Images Using Conditional Generative Adversarial Networks, 2017, IEEE International Conference on Computer Vision Workshops (ICCVW)… [cited by applicant]
Response to office action dated Apr. 21, 2023 in Japanese Patent Application No. 2020-552396, (13 pages). [cited by applicant]
Yan Zhang et al., High-Throughput, Label-Free and Slide-Free Histological Imaging by Computational Microscopy and Unsupervised Learning, Adv. Sci. 2022, 9, 2102358 (14 pages). [cited by applicant]
First Examination Report dated of Feb. 27, 2023 for European Patent Application No. 19776693.4, (7 pages). [cited by applicant]
Sinno Jialin Pan et al., A Survey on Transfer Learning, IEEE Transactions on Knowledge and Data Engineering, vol. 22, No. 10, 1345-1359 (Oct. 2010). [cited by applicant]
Response to First Examination Report dated of Jul. 10, 2023 for European Patent Application No. 19776693.4, (16 pages). [cited by applicant]
Office Action dated May 9, 2023 for U.S. Appl. No. 17/261,542, (21 pages). [cited by applicant]
George Barbastathis et al., On the use of deep learning for computation imaging, Optica, Jul. 25, 2019, pp. 921-943. [cited by applicant]
George Barbastathis et al., On the use of deep learning for computation imaging, Optica, Aug. 2019. [cited by applicant]
YoungJu Jo et al., Quantitative Phase imaging and Artificial Intelligence: A Review, IEEE J. of Selected Topics in Quantum Electronics, vol. 25, No. 1, Jan./Feb. 2019 (14 pages). [cited by applicant]
Response to Office Action dated Aug. 9, 2023 for U.S. Appl. No. 17/261,542, (9 pages). [cited by applicant]
Office Action dated Nov. 9, 2023 for U.S. Appl. No. 17/261,542, (12 pages). [cited by applicant]
Notification of the First Office Action dated Nov. 4, 2023, for Chinese Patent Application No. 2019800291722, (40 pages). [cited by applicant]
Notice of Allowance and Fee(s) Due dated Jul. 23, 2024 for U.S. Appl. No. 17/261,542, (16 pages). [cited by applicant]
Notification of the First Office Action dated Nov. 4, 2023 for Chinese Appl No. 2019800291722, (40 pages). [cited by applicant]
Response to the First Office Action dated May 20, 2024 for Chinese Appl No. 2019800291722; English Translations for Claims only (18 pages). [cited by applicant]
Notification of the Second Office Action dated May 25, 2024 for Chinese Appl No. 2019800291722, (40 pages). [cited by applicant]
Johannes Schindelin et al., Fiji: an open-source platform for biological-image analysis, Nature Methods, (Jul. 2012), vol. 9, No. 7, 676-682. [cited by applicant]
Liang Han et al., Transferring Microscopy Image Modalities with Conditional Generative Adversarial Networks, 2017, IEEE Conference on Computer Vision and Pattern Recognition Workshops. [cited by applicant]
Notice of Preliminary Rejection dated Dec. 13, 2024 for Korean Patent Application No. 10-2020-7031116, (18 pages). [cited by applicant]
Response to the Second Office Action dated Sep. 24, 2024 for Chinese Appl No. 2019800291722, English translations for pending claims only (16 pages). [cited by applicant]
Notice of Allowance dated Nov. 25, 2024 for Chinese Appl No. 2019800291722, (4 pages). [cited by applicant]
Cited By (1)
US 12,700,249