IP Library › Granted Patent US 12,682,213
Granted Patent B2
US 12,682,213 · App. 17/431,871 · Granted Jul 14, 2026

Identifying microorganisms using three-dimensional quantitative phase imaging

Inventors: Kihyun Hong (Daejeon, KR); Hyun-Seok Min (Daejeon, KR); YongKeun Park (Daejeon, KR); Geon Kim (Daejeon, KR); Youngju Jo (Daejeon, KR)
Assignee: Tomocube, Inc.
G06N3/045G06T7/0012G06T2207/10056
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,682,213
App. No.
17/431,871
Granted
Jul 14, 2026
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying the predicted type of one or more microorganisms. In one aspect, a system comprises a phase-contrast microscope and a microorganism classification system. The phase-contrast microscope is configured to generate a three-dimensional quantitative phase image of one or more microorganisms. The microorganism classification system is configured to process the three-dimensional quantitative phase image using a neural network to generate a neural network output characterizing the microorganisms, and thereafter identify the predicted type of the microorganisms using the neural network output.

Claims (54)

1 . A method comprising:

using a phase-contrast microscope to generate a three-dimensional (3D) quantitative phase image (QPI) of one or more microorganisms, wherein the 3D QPI comprises a 3D representation of the microorganisms; and

generating a latent feature representation of the microorganisms using a neural network, comprising:

processing the 3D QPI using the neural network to generate the latent feature representation of the microorganisms as an intermediate output generated by one or more intermediate layers of the neural network, wherein each intermediate layer is a hidden layer of the neural network, wherein:

the neural network is a convolutional neural network comprising one or more 3D convolutional layers; and

the neural network has been trained to process an input 3D QPI in accordance with current values of a set of neural network parameters of the neural network to generate a neural network output characterizing the microorganisms;

the neural network has been trained perform a plurality of different prediction tasks, including two or more of: predicting a species of microorganisms in the input 3D QPI, predicting a strain of microorganisms in the input 3D QPI, predicting a gram-stainability of microorganisms in the input 3D QPI, predicting a metabolism of microorganisms in the input 3D QPI, predicting a morphology of microorganisms in the input 3D QPI, or predicting a motility of microorganisms in the input 3D QPI; and

processing the latent feature representation of the microorganisms using a separate prediction machine learning model that is different from the neural network to generate a prediction characterizing the microorganisms.

2 . The method of claim 1 , wherein:

the neural network output comprises a respective probability value for each of a predetermined number of microorganism types; and

the probability value for a given microorganism type indicates a likelihood that the microorganisms are of the given microorganism type.

3 . The method of claim 1 , wherein the neural network output comprises a probability value of a microorganism type that indicates a likelihood that the microorganisms are of the microorganism type.

4 . The method of claim 1 , wherein the predicted type of the microorganisms is selected from the group consisting of genus, species, strain, gram-stainability, metabolism, morphology, and motility.

5 . The method of claim 1 , wherein the three-dimensional quantitative phase image of the microorganisms is a three-dimensional refractive index tomogram.

6 . The method of claim 1 , wherein the microorganisms are selected from the group consisting of bacteria, viruses, fungi, parasites, and microalgae.

7 . The method of claim 6 , wherein the microorganisms comprise bacteria.

8 . The method of claim 7 , wherein the bacteria are present in a blood sample of a patient.

9 . The method of claim 8 , further comprising administering an antibiotic therapy to the patient based on the predicted type of the bacteria.

10 . The method of claim 1 , wherein using a phase-contrast microscope to generate a three-dimensional quantitative phase image of the microorganisms comprises:

using the phase-contrast microscope to generate phase and amplitude images of the microorganisms at each of a plurality of illumination angles; and

reconstructing a three-dimensional refractive index tomogram using the phase and amplitude images.

11 . The method of claim 1 , wherein the method takes at most one hour.

12 . The method of claim 1 , wherein:

the one or more microorganisms are isolated from a biological sample from a patient with a bacterial infection or that is suspected of having a bacterial infection; and

the prediction characterizing the microorganisms is generated within one hour of obtaining the biological sample from the patient.

13 . The method of claim 12 , wherein the biological sample comprises a blood sample.

14 . The method of claim 12 , wherein the prediction characterizing the microorganisms is generated within 45 minutes of obtaining the biological sample from the patient.

15 . The method of claim 12 , wherein the prediction characterizing the microorganisms is generated within 30 minutes of obtaining the biological sample from the patient.

16 . The method of claim 12 , wherein the prediction characterizing the microorganisms is generated within 15 minutes of providing the biological sample from the patient.

17 . One or more non-transitory computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving a three-dimensional (3D) quantitative phase image (QPI) of one or more microorganisms that is generated using a phase-contrast microscope, wherein the 3D QPI comprises a 3D representation of the microorganisms; and

generating a latent feature representation of the microorganisms using a neural network, comprising:

processing the 3D QPI using the neural network to generate the latent feature representation of the microorganisms as an intermediate output generated by one or more intermediate layers of the neural network, wherein each intermediate layer is a hidden layer of the neural network, wherein:

the neural network is a convolutional neural network comprising one or more 3D convolutional layers; and

the neural network has been trained to process an input 3D QPI in accordance with current values of a set of neural network parameters of the neural network to generate a neural network output characterizing the microorganisms;

the neural network has been trained perform a plurality of different prediction tasks, including two or more of: predicting a species of microorganisms in the input 3D QPI, predicting a strain of microorganisms in the input 3D QPI, predicting a gram-stainability of microorganisms in the input 3D QPI, predicting a metabolism of microorganisms in the input 3D QPI, predicting a morphology of microorganisms in the input 3D QPI, or predicting a motility of microorganisms in the input 3D QPI; and

processing the latent feature representation of the microorganisms using a separate prediction machine learning model that is different from the neural network to generate a prediction characterizing the microorganisms.

18 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving a three-dimensional (3D) quantitative phase image (QPI) of one or more microorganisms that is generated using a phase-contrast microscope, wherein the 3D QPI comprises a 3D representation of the microorganisms; and

generating a latent feature representation of the microorganisms using a neural network, comprising:

processing the 3D QPI using the neural network to generate the latent feature representation of the microorganisms as an intermediate output generated by one or more intermediate layers of the neural network, wherein each intermediate layer is a hidden layer of the neural network, wherein:

the neural network is a convolutional neural network comprising one or more 3D convolutional layers; and

the neural network has been trained to process an input 3D QPI in accordance with current values of a set of neural network parameters of the neural network to generate a neural network output characterizing the microorganisms;

the neural network has been trained perform a plurality of different prediction tasks, including two or more of: predicting a species of microorganisms in the input 3D QPI, predicting a strain of microorganisms in the input 3D QPI, predicting a gram-stainability of microorganisms in the input 3D QPI, predicting a metabolism of microorganisms in the input 3D QPI, predicting a morphology of microorganisms in the input 3D QPI, or predicting a motility of microorganisms in the input 3D QPI; and

processing the latent feature representation of the microorganisms using a separate prediction machine learning model that is different from the neural network to generate a prediction characterizing the microorganisms.

19 . The method of claim 1 , wherein the prediction machine learning model is a non-differentiable machine learning model.

20 . The method of claim 1 , wherein the prediction machine learning model comprises a random forest model.

21 . The method of claim 1 , wherein the prediction machine learning model comprises a support vector machine model.

22 . The method of claim 1 , wherein the one or more intermediate layers of the neural network that generate the latent feature representation of the microorganisms comprise at least one fully-connected intermediate layer of the neural network.

23 . The system of claim 18 , wherein:

the neural network output comprises a respective probability value for each of a predetermined number of microorganism types; and

the probability value for a given microorganism type indicates a likelihood that the microorganisms are of the given microorganism type.

Assignments (5)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 26, 2023
From: KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY
To: TOMOCUBE, INC.
Reel/Frame 062500/0814 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2022
From: PARK, YONGKEUN; KIM, GEON; JO, YOUNGJU
To: KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY
Reel/Frame 060624/0155 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2022
From: PARK, YONGKEUN; KIM, GEON; JO, YOUNGJU
To: KOREA ADVANCED INSTITUTE OF SCIENCE AND TECHNOLOGY
Reel/Frame 060624/0176 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2022
From: HONG, KIHYUN; MIN, HYUN-SEOK
To: TOMOCUBE, INC.
Reel/Frame 060624/0306 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2022
From: HONG, KIHYUN; MIN, HYUN-SEOK
To: TOMOCUBE, INC.
Reel/Frame 060919/0956 →
Continuity (3)
Provisional Application 62856290 · Jun 3, 2019
Provisional Application 62817680 · Mar 13, 2019
Related Publication 20220156561A1 · May 19, 2022
References Cited (98)
US 9436895B1 · Jones et al. · 2016 [cited by applicant]
US 9971966B2 · Nelson et al. · 2018 [cited by applicant]
US 10082653B1 · Liu et al. · 2018 [cited by applicant]
US 12001940B2 · Hong et al. · 2024 [cited by applicant]
US 20160244834A1 · Ong · 2016 [cited by examiner]
US 20170032285A1 · Sharma et al. · 2017 [cited by applicant]
US 20170249548A1 · Nelson et al. · 2017 [cited by applicant]
US 20170261930A1 · Mathuis et al. · 2017 [cited by applicant]
US 20170357084A1 · Park et al. · 2017 [cited by applicant]
US 20180157966A1 · Henry et al. · 2018 [cited by applicant]
US 20180322634A1 · Zimmerman et al. · 2018 [cited by applicant]
US 20190056296A1 · Lee et al. · 2019 [cited by applicant]
US 20200208191A1 · Galiano · 2020 [cited by examiner]
US 20200394794A1 · Park et al. · 2020 [cited by applicant]
US 20210056687A1 · Hibbard et al. · 2021 [cited by applicant]
US 20210303818A1 · Randolph · 2021 [cited by applicant]
US 20220180515A1 · Zhou et al. · 2022 [cited by applicant]
US 20220383562A1 · Park et al. · 2022 [cited by applicant]
US 20220383986A1 · Popescu et al. · 2022 [cited by applicant]
US 20230013209A1 · Hong et al. · 2023 [cited by applicant]
CN 103827889A · 2014 [cited by applicant]
CN 108885682 · 2018 [cited by applicant]
JP 2017520864 · 2017 [cited by applicant]
JP 2018506788 · 2018 [cited by applicant]
JP 201892560 · 2018 [cited by applicant]
JP 2019506208 · 2019 [cited by applicant]
WO WO2017053592 · 2017 [cited by applicant]
WO WO2017216123A1 · 2017 [cited by applicant]
WO WO2019002602 · 2019 [cited by applicant]
WO WO2019018693 · 2019 [cited by applicant]
WO WO2019032723 · 2019 [cited by applicant]
Lee et al., Quantitative Phase Imaging Techniques for the Study of Cell Pathophysiology: From Principles to Applications, Feb. 19, 2013, Sensors 13(4), 4170-4191 (Year: 2013). [cited by examiner]
Jo et al., Holographic deep learning for rapid optical screening of anthrax spores, Feb. 27, 2017, American Association for the Advancement of Science, 2375-2548 (Year: 2017). [cited by examiner]
Suk, HI., Lee, SW., Shen, D. et al. Latent feature representation with stacked auto-encoder for AD/MCI diagnosis. Brain Struct Funct 220, 841-859 (2015). (Year: 2015). [cited by examiner]
Thomas Mortier, et. al., Bacterial species identification using MALDI-TOF mass spectrometry and machine learning techniques: A large-scale benchmarking study, Computational and Structural Biotechnology Journal (Year: 20… [cited by examiner]
G. Kim et al., “Automated Identification of Bacteria using Three-Dimensional Holographic Imaging and Convolutional Neural Network,” 2018 IEEE Photonics Conference (IPC), Reston, VA, USA, 2018, pp. 1-2, doi: 10.1109/IPCo… [cited by examiner]
Extended European Search Report issued in European Application No. 19919488.7 on Jan. 31, 2022, 9 pages. [cited by applicant]
Kim et al., “Automated Identification of Bacteria Using Three-Dimensional Holographic Imaging and Convolutional Neural Network,” 2018 IEEE Photonics Conference (IPC), Sep. 2018, 2 pages. [cited by applicant]
Nguyen et al., “Gradient light interference microscopy for 3D imaging of unlabeled specimens,” Nat. Commun., Aug. 2017, 8:210, 9 pages. [cited by applicant]
Japanese Notice of Reasons for Refusal, with machine-generated translation, for corresponding JP Appl No. 2021-555105, dated Aug. 29, 2023. [cited by applicant]
YoungJu Jo et al., Quantitative Phase Imaging and Artificial Intelligence: A Review, IEEE Journal of Selected Topics in Quantum Electronics, vol. 25, No. 1, 2018, 6800914 (p. 1 /14), DOI:10.1109/JSTQE.2018.2859234. [cited by applicant]
Adhikari et al., “Critical care and the global burden of critical illness in adults,” The Lancet, Oct. 2010, 376(9749): 1339-1346. [cited by applicant]
Akhi et al., “Antibiotic susceptibility pattern of aerobic and anaerobic bacteria isolated from surgical site infection of hospitalized patients,” Jundishapur Journal of Microbiology, Jul. 2015, 8(7): 6 pages. [cited by applicant]
Anand et al., “Automated Disease Identification With 3-D Optical Imaging: A Medical Diagnostic Tool,” Proceedings of the IEEE, Mar. 2017, 105(5): 924-946. [cited by applicant]
Charriere et al., “Cell refractive index tomography by digital holographic microscopy,” Optics Letters, Jan. 2006, 31(2): 178-180. [cited by applicant]
Chattopadhyay et al., “Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks,” 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar. 2018, 839-847. [cited by applicant]
Chen et al., “Deep Learning in Label-free Cell Classification,” Scientific Reports, Mar. 2016, 6(1), 6 pages. [cited by applicant]
Claydon et al., “The rapid identification of intact microorganisms using mass spectrometry,” Nature Biotechnology, Nov. 1996, 14(11): 1584-1586. [cited by applicant]
Cleven et al., “Identification and characterization of bacterial pathogens causing bloodstream infections by DNA microarray,” Journal of Clinical Microbiology, Jul. 2006, 44(7): 2389-2397. [cited by applicant]
Ferreira et al. “Direct identification of urinary tract pathogens from urine samples by matrix-assisted laser desorption ionization-time of flight mass spectrometry,” Journal of Clinical Microbiology, Jun. 2010, 48(6): … [cited by applicant]
Fonkwo, “Pricing infectious disease: The economic and health implications of infectious diseases,” EMBO reports, Jul. 2008, 9(S1): S13-17. [cited by applicant]
Gallanos et al., “Preparation and properties of antisera against the lipid-A component of bacterial lipopolysaccharides,” European Journal of Biochemistry, Dec. 1971, 24(1): 116-122. [cited by applicant]
Hoentjen et al., “Antibiotics with a selective aerobic or anaerobic spectrum have different therapeutic activities in various regions of the colon in interleukin 10 gene deficient mice,” Gut, Dec. 2003, 52(12): 1721-172… [cited by applicant]
Huang et al., “Densely connected convolutional networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 4700-4708. [cited by applicant]
Järvinen et al., “Rapid identification of bacterial pathogens using a PCR- and microarray-based assay,” BMC Microbiology, Dec. 2009, 9(1): 6 pages. [cited by applicant]
Javidi et al., “Three-dimensional imaging and recognition of microorganism using single-exposure on-line (SEOL) digital holography,” Opt Express, Jun. 2005, 13(12): 4492-4506. [cited by applicant]
Jo et al., “Angle-resolved light scattering of individual rod-shaped bacteria based on Fourier transform light scattering,” Scientific Reports, May 2014, 4(1): 1-6. [cited by applicant]
Jo et al., “Holographic deep learning for rapid optical screening of anthrax spores,” Science Advances, Aug. 2017, 3(8), 10 pages. [cited by applicant]
Jo et al., “Label-free identification of individual bacteria using Fourier transform light scattering,” Opt Express, Jun. 2015, 23(12): 15792-15805. [cited by applicant]
Jo et al., “Quantitative Phase Imaging and Artificial Intelligence: A Review,” IEEE Journal of Selected Topics in Quantum Electronics, Aug. 2018, 25(1): 1-4. [cited by applicant]
Kang et al., “Bloodstream infections caused by antibiotic-resistant gram-negative bacilli: risk factors for mortality and impact of inappropriate initial antimicrobial therapy on outcome,” Antimicrobial Agents and Chemo… [cited by applicant]
Kim et al., “Correlative three-dimensional fluorescence and refractive index tomography: bridging the gap between molecular specificity and quantitative bioimaging,” Biomedical Optics Express, Dec. 2017, 8(12): 5688-569… [cited by applicant]
Kim et al., “Learning-based screening of hematologic disorders using quantitative phase imaging of individual red blood cells,” Biosensors and Bioelectronics, Jan. 2019, 123: 69-76. [cited by applicant]
Kim et al., “High-resolution three-dimensional imaging of red blood cells parasitized by Plasmodium falciparum and in situ hemozoin crystals using optical diffraction tomography,” Journal of Biomedical Optics, Jun. 2013… [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, Dec. 2014, 15 pages. [cited by applicant]
Lai et al., “A DenseNet-based diagnosis algorithm for automated diagnosis using clinical ECG data,” Journal of Southern Medical University, Jan. 2019, 39(1): 69-75 (with English Abstract). [cited by applicant]
Lecun et al., “Deep learning,” Nature, May 2015, 521(7553): 436-444. [cited by applicant]
Li et al., “Imaging through glass diffusers using densely connected convolutional networks,” Optica, Jul. 2018, 5(7): 803-813. [cited by applicant]
Liu et al., “Hospital deaths in patients with sepsis from 2 independent cohorts,” Jama, Jul. 2014, 312(1): 90-92. [cited by applicant]
Morata et al., “Influence of multidrug resistance and appropriate empirical therapy on the 30-day mortality rate of Pseudomonas aeruginosa bacteremia,” Antimicrobial Agents and Chemotherapy, Sep. 2012, 56(9): 4833-4837. [cited by applicant]
Morens et al., “Emerging infectious diseases: threats to human health and global stability,” PLoS pathogens, Jul. 2013, 9(7): 3 pages. [cited by applicant]
Nadkarni et al., “Determination of bacterial load by real-time PCR using a broad-range (universal) probe and primers set,” Microbiology, Jan. 2002, 148(1): 257-266. [cited by applicant]
Negut et al., “Experimental approach for bacteriophage susceptibility testing of planktonic and sessile bacterial populations—Study protocol,” Germs, Dec. 2014, 4(4): 92. [cited by applicant]
Paul et al., “Systematic review and meta-analysis of the efficacy of appropriate empiric antibiotic therapy for sepsis,” Antimicrobial Agents and Chemotherapy, Nov. 2010, 54(11): 4851-4863. [cited by applicant]
PCT International Search Report and Written Opinion issued in International Application No. PCT/IB2019/058248 on Jan. 9, 2020, 11 pages. [cited by applicant]
Pitt et al., “Rapid separation of bacteria from blood-review and outlook,” Biotechnology Progress, Jul. 2016, 32(4): 823-839. [cited by applicant]
Ranjbar et al., “DNA microarray for rapid detection and identification of food and water borne Bacteria: from dry to wet lab,” The Open Microbiology Journal, Nov. 2017, 11: 330-338. [cited by applicant]
Rice, “Antimicrobial resistance in gram-positive bacteria,” American Journal of Infection Control, Jun. 2006, 34(5): S11-S19. [cited by applicant]
Shin et al., “Active illumination using a digital micromirror device for quantitative phase imaging,” Optics Letters, Nov. 2015, 40(22): 5407-5410. [cited by applicant]
Smith et al., “Global rise in human infectious disease outbreaks,” Journal of the Royal Society Interface, Dec. 2014, 11(101): 6 pages. [cited by applicant]
Van der Maaten et al., “Visualizing Data using t-SNE,” Journal of Machine Learning Research, Nov. 2008, 9(11): 27 pages. [cited by applicant]
Vercruysse et al., “Three-part differential of unlabeled leukocytes with a compact lens-free imaging flow cytometer,” Lab Chip, 2015, 15(4): 1123-1132. [cited by applicant]
Vouga et al., “Emerging bacterial pathogens: the past and beyond,” Clinical Microbiology and Infection, Jan. 2016, 22(1): 12-21. [cited by applicant]
Walsh et al., “Activity and mechanisms of action of selected biocidal agents on Gram-positive and-negative bacteria,” Journal of Applied Microbiology, Feb. 2003, 94(2): 240-247. [cited by applicant]
Wu et al., “The social impact of pathogen threat: How disease salience influences conformity,” Personality and Individual Differences , Jul. 2012, 53(1): 50-54. [cited by applicant]
Yoon et al., “Identification of non-activated lymphocytes using three-dimensional refractive index tomography and machine learning,” Scientific Reports, Jul. 2017, 7(1): 1-10. [cited by applicant]
Zhou et al., “Holistic brain tumor screening and classification based on densenet and recurrent neural network,” International MICCAI Brainlesion Workshop, Sep. 2018, 10 pages. [cited by applicant]
PCT International Preliminary Report on Patentability issued in International Application No. PCT/IB2019/058248 on Sep. 23, 2021, 10 pages. [cited by applicant]
Japanese Notice of Reasons for Refusal, with formal translation, for corresponding JP Appl No. 2021-555105, dated Aug. 29, 2023. [cited by applicant]
Chiritescu et al., “3D Imaging of Live Specimens Quantitative Phase Imaging with Slim and Glim,” Imaging & Microscopy, Apr. 2018, 9 pages. [cited by applicant]
Jung et al., “Label-free non-invasive quantitative measurement of lipid contents in individual microalgal cells using refractive index tomography,” Sci. Rep., Apr. 2018, 8(1):6524, 18 pages. [cited by applicant]
Lee et al., “Quantitative phase imaging techniques for the study of cell pathophysiology: from principles to applications,” Sensors, Mar. 2013, 13(4):4170-4191. [cited by applicant]
Çiçek et al., “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation,” CoRR, submitted on Jun. 2016, arXiv:1606.06650, 8 pages. [cited by applicant]
Lu et al., “A 3D Convolutional Neural Network for Volumetric Image Semantic Segmentation,” In Proceedings of the 43rd North American Manufacturing Research Conference (NAMRC 43), Jun. 8-12, 2015, North Carolina, USA, 39… [cited by applicant]
Qi et al., “Volumetric and Multi-view CNNs for Object Classification on 3D Data,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, Jun. 27-30, 2016, 5648-5656. [cited by applicant]
Japanese Notice of Reasons for Refusal, for corresponding JP Appl. No. 2021-555105, dated Jan. 23, 2024, 12 pages (with English translation). [cited by applicant]
Notice of Allowance in Chinese Appln. No. 201980093334.9, dated Dec. 5, 2024, 12 pages (with English translation). [cited by applicant]
Hay et al., “Performance of convolutional neural networks for identification of bacteria in 3D microscopy datasets,” Computational Biology, Dec. 2018, 14(12):e1006628, 17 pages. [cited by applicant]