IP Library › Granted Patent US 12,190,478
Granted Patent B2
US 12,190,478 · App. 17/530,471 · Granted Jan 7, 2025

Systems and methods for deep learning microscopy

Inventors: Aydogan Ozcan (Los Angeles, CA); Yair Rivenson (Los Angeles, CA); Hongda Wang (Los Angeles, CA); Harun Gunaydin (Los Angeles, CA); Kevin de Haan (Los Angeles, CA)
Assignee: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
G06T5/50G06N3/08G06T3/4046G06T3/4076G06T5/70G06T5/73G06T5/92G06T7/0014G06T2207/10056G06T2207/10064G06T2207/10101G06T2207/20081G06T2207/20084G06T2207/30024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,478
App. No.
17/530,471
Granted
Jan 7, 2025
Kind
B2
Abstract

A microscopy method includes a trained deep neural network that is executed by software using one or more processors of a computing device, the trained deep neural network trained with a training set of images comprising co-registered pairs of high-resolution microscopy images or image patches of a sample and their corresponding low-resolution microscopy images or image patches of the same sample. A microscopy input image of a sample to be imaged is input to the trained deep neural network which rapidly outputs an output image of the sample, the output image having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast.

Claims (13)

1. A system for outputting microscopy images of a test sample comprising a computing device having image processing software executed thereon, the image processing software comprising a trained deep neural network that is executed using one or more processors of the computing device, wherein the trained deep neural network is trained with a training set of non-fluorescence images comprising co-registered pairs of high-resolution microscopy images or image patches of a training sample and their corresponding low-resolution microscopy images or image patches of the same training sample, the image processing software configured to receive a non-fluorescence microscopy input image of the test sample comprising one of a bright-field microscopy image, a holographic microscopy image, a confocal microscopy image, a dark-field microscopy image, or an optical coherence tomography (OCT) image and output an output image of the test sample having improved one or more of spatial resolution, depth-of-field, signal-to-noise ratio, and/or image contrast.

2. The system of claim 1 , further comprising a microscope configured to generate the non-fluorescence microscopy input image of the test sample.

3. The system of claim 2 , wherein the microscope comprises a confocal microscope.

4. The system of claim 1 , wherein the trained deep neural network comprises a trained convolutional neural network (CNN) that is a GAN-trained network.

5. The system of claim 1 , wherein the computing device comprises a personal computer, laptop, server, mobile computing device, one or more graphics processing units (GPUs), or application specific integrated circuit (ASIC).

6. The system of claim 1 , wherein the trained deep neural network outputs the output image of the test sample within one second of inputting the non-fluorescence microscopy input image to the trained deep neural network.

7. The system of claim 1 , further comprising a monitor or display.

8. The system of claim 1 , wherein the non-fluorescence microscopy input image of the test sample comprises a histopathological slide image.

9. The system of claim 1 , wherein the training sample comprises tissue of a first type and wherein the test sample also comprises a tissue sample of the first type.

10. The system of claim 1 , wherein the training sample comprises tissue of a first type and wherein the test sample comprises a tissue sample of the second type different from the first type.

11. The system of claim 1 , wherein the training sample comprises tissue stained with a stain or dye of a first type and wherein the test sample also comprises a tissue sample stained with a stain or dye of a first type.

12. The system of claim 1 , wherein the training sample comprises tissue stained with a stain or dye of a first type and wherein the test sample comprises a tissue sample stained with a stain or dye of a second type different from the first type.

13. The system of claim 1 , wherein the high-resolution microscopy images or image patches of a training sample are obtained by synthesizing a higher resolution image from multiple, sub-pixel shifted low-resolution images.

Continuity (6)
Division 16395674 · Apr 26, 2019
Provisional Application 62798336 · Jan 29, 2019
Provisional Application 62698581 · Jul 16, 2018
Provisional Application 62670612 · May 11, 2018
Provisional Application 62662943 · Apr 26, 2018
Related Publication 20220114711A1 · Apr 14, 2022
References Cited (64)
US 10824847B2 · Wu · 2020 [cited by examiner]
US 11222415B2 · Ozcan · 2022 [cited by examiner]
US 11676247B2 · Zimmer · 2023 [cited by examiner]
US 11776124B1 · Behrooz · 2023 [cited by examiner]
US 20120099803A1 · Ozcan et al. · 2012 [cited by applicant]
US 20120148141A1 · Ozcan et al. · 2012 [cited by applicant]
US 20120157160A1 · Ozcan et al. · 2012 [cited by applicant]
US 20120218379A1 · Ozcan et al. · 2012 [cited by applicant]
US 20120248292A1 · Ozcan et al. · 2012 [cited by applicant]
US 20120281899A1 · Ozcan et al. · 2012 [cited by applicant]
US 20130092821A1 · Ozcan et al. · 2013 [cited by applicant]
US 20130157351A1 · Ozcan et al. · 2013 [cited by applicant]
US 20130193544A1 · Ozcan et al. · 2013 [cited by applicant]
US 20130203043A1 · Ozcan et al. · 2013 [cited by applicant]
US 20130258091A1 · Ozcan et al. · 2013 [cited by applicant]
US 20130280752A1 · Ozcan et al. · 2013 [cited by applicant]
US 20140120563A1 · Ozcan et al. · 2014 [cited by applicant]
US 20140160236A1 · Ozcan et al. · 2014 [cited by applicant]
US 20140300696A1 · Ozcan et al. · 2014 [cited by applicant]
US 20150111201A1 · Ozcan et al. · 2015 [cited by applicant]
US 20150153558A1 · Ozcan et al. · 2015 [cited by applicant]
US 20150204773A1 · Ozcan et al. · 2015 [cited by applicant]
US 20150357083A1 · Ozcan et al. · 2015 [cited by applicant]
US 20160070092A1 · Ozcan et al. · 2016 [cited by applicant]
US 20160161409A1 · Ozcan et al. · 2016 [cited by applicant]
US 20160327473A1 · Ozcan et al. · 2016 [cited by applicant]
US 20160334614A1 · Ozcan et al. · 2016 [cited by applicant]
US 20170153106A1 · Ozcan et al. · 2017 [cited by applicant]
US 20170160197A1 · Ozcan · 2017 [cited by examiner]
US 20170168285A1 · Ozcan · 2017 [cited by examiner]
US 20170220000A1 · Ozcan et al. · 2017 [cited by applicant]
US 20180003686A1 · Ozcan et al. · 2018 [cited by applicant]
US 20180052425A1 · Ozcan et al. · 2018 [cited by applicant]
US 20180196193A1 · Ozcan et al. · 2018 [cited by applicant]
US 20180293707A1 · El-Khamy · 2018 [cited by examiner]
US 20180373921A1 · Di Carlo et al. · 2018 [cited by applicant]
US 20200250794A1 · Zimmer · 2020 [cited by examiner]
US 20220012850A1 · Ozcan · 2022 [cited by examiner]
Abadi, Martin et al., TensorFlow: A system for large-scale machine learning, https://github.com/tensorflow/tensorflow, arXiv:1605.08695v2 [cs.DC] Mar. 31, 2016, pp. 1-18. [cited by applicant]
Bishara, Waheb et al., Lensfree on-chip microscopy over a wide field-of-view using pixel super-resolution, Optics Expresss, May 24, 2018, vol. 18, No. 11, 11181-11191. [cited by applicant]
Farsiu, Sina et al., Fast and Robust Multiframe Super Resolution, IEEE Transactions on Image Processing, vol. 13, No. 10, Oct. 2004, pp. 1327-1344. [cited by applicant]
Goodfellow, Ian J. et al., Generative Adversarial Nets, date: unknown; http://www.github.com/goodfell/adversarial (9 pages). [cited by applicant]
Greenbaum, Alon et al., Maskless imaging of dense samples using pixel super-resolution based multi-height lensfree on-chip microscopy, Jan. 30, 2012, vol. 20, No. 3, Optics Express, pp. 3129-3143. [cited by applicant]
Hayat, Khizar et al., Super-Resolution via Deep Learning, arXiv:1706.09077v1 [cs.CV[ Jun. 28, 2017 (33 pages). [cited by applicant]
He, Kaiming et al., Deep Residual Learning for Image Recognition, http://image-net.org/challenges/LSVRC/2015/ and http://mscoco.org/dataset/#detections-challenge20158, pp. 770-778 (2016). [cited by applicant]
Huang, Xiwei et al., Machine Learning Based Single-Frame Super-Resolution Processing for Lensless Blood Cell Counting, Sensors 2016, 16, 1836; doi:10.3390/s16111836, www.mdpi.com/journal/sensors. [cited by applicant]
Kingma, Diederik P. et al., Adam: A Method for Stochastic Optimization, Published as a conference paper at ICLR 2015, arXiv:1412.6980v9 [cs.LG] Jan. 30, 2017, pp. 1-15. [cited by applicant]
Ledig, Christian et al., Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, Computer Vision Foundation, date: unknown, pp. 4681-4690. [cited by applicant]
Li, Dong et al., Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics, Science. Aug. 28, 2015; 346(6251):aab3500. doi:10.1126/science.aab3500 (26 pages). [cited by applicant]
Liu, Tairan et al., Deep Learning-based super-resolution in coherent imaging systems, Scientific Reports (2019) 9:3926; https://doi.org/10.1038/s41598-019-40554-1 (13 pages). [cited by applicant]
Luo, Wei et al., Propagation phasor approach for holographic image reconstruction, Scientific Reports, 6:22738, DOI:10.1038/srep22738, pp. 1-14. [cited by applicant]
Rivenson, Yair et al., Deep learning-based virtual histology staining using auto-fluorescence of label-free tissue, arXiv:1803.11293 [cs.CV], Apr. 2, 2018 (22 pages). [cited by applicant]
Rivenson, Yair et al., Phase recovery and holographic image reconstruction using deep learning in neural networks, ArXiv170504286 Phys. (2017) (30 pages). [cited by applicant]
Rivenson, Yair et al., Deep learning microscopy, vol. 4, No. 11, Nov. 2017, Optica, pp. 1437-1443. [cited by applicant]
Ronneberger, Olaf et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, arXiv:1505.04597v1 [cs.CV] May 18, 2015, pp. 1-8. [cited by applicant]
Shimobaba, Tomoyoshi et al., Convolutional neural network-based regression for depth prediction in digital holography, arXiv:1802.00664v1 [cs.CV] Feb. 2, 2018, pp. 1-4. [cited by applicant]
Sinha, Ayan et al., Lensless computational imaging through deep learning, vol. 4, No. 9, Sep. 2017, Optica, pp. 1117-1125. [cited by applicant]
Jin, Kyong Hwan et al., Deep Convolutional Neural Network for Inverse Problems in Imaging, IEEE Transactions on Image Processing, vol. 26, No. 9, Sep. 2017, 4509-4522. [cited by applicant]
Yang, Wenming et al., Deep Learning for Single Image Super-Resolution: A Brief Review, arXiv:1808.03344v1 [cs.CV] Aug. 9, 2018 (15 pages). [cited by applicant]
Willy, N.M. et al., Membrane mechanics govern spatiotemporal heterogeneity of endocytic clathrin coat dynamics, Molecular Biology of the Cell (2017), 3480-3488. [cited by applicant]
Wu, Yichen et al., Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery, vol. 5, No. 6, Jun. 2018, Optica, pp. 704-710. [cited by applicant]
Zhang, Yibo et al., Accurate color imaging of pathology slides using holography and absorbance spectrum estimation of histochemical stains, J. Biophotonics. 2019; 12:e201800335. [cited by applicant]
Heinrich, Larissa et al., Deep Learning for Isotropic Super-Resolution from Non-isotropic 3D Electron Microscopy, MICCAI 2017, Part II, LNCS 10434, pp. 135-143 (2017) doi:10.1007/978-3-319-66185-8_16. [cited by applicant]
Weigert, Martin et al., Isotropic Reconstruction of 3D Fluorescence Microscopy Images Using Convolutional Neural Networks, MICCAI 2017, Part II, LNCS 10434, pp. 126-134 (2017) doi: 10.1007/978-3-319-66185-8_15. [cited by applicant]
Cited By (4)
US 12,373,916 US 12,400,291 US 12,430,713 US 12,548,113