IP Library › Granted Patent US 12,254,624
Granted Patent B2
US 12,254,624 · App. 17/547,033 · Granted Mar 18, 2025

Artificial intelligence enabled reagent-free imaging hematology analyzer

Inventors: Renjie Zhou (Hong Kong, CN); Xin Shu (Hong Kong, CN); Rishikesh Pandey (Unionville, CT)
Assignee: The Chinese University of Hong Kong
G06T7/0012G06N3/045G06V10/778G06V10/94G06V30/18057G06V30/1912G06V30/19127G06T2207/20081G06T2207/20084G06T2207/30024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,254,624
App. No.
17/547,033
Granted
Mar 18, 2025
Kind
B2
Abstract

The subject invention pertains to methods and systems for classifying leukocytes using artificial intelligence called AIRFIHA (artificial-intelligence enabled reagent-free imaging hematology analyzer) that can accurately classify subpopulations of leukocytes in a label-free manner. AIRFIHA can not only subtype lymphocytes into B and T cell but is capable of sorting different types of T cells subtypes. AIRFIHA is realized through training a two-step neural network using label-free images of separated leukocytes acquired from a custom-built quantitative phase microscope. Owing to its easy operation, low cost, and strong discerning capability of complex leukocyte subpopulations, AIRFIHA is clinically translatable and can also be deployed in resource-limited settings.

Claims (40)

1. A method of classifying information of leukocytes in a sample, the method comprising:

training a neural network according to parameters derived from digitally observed features of known leukocytes;

obtaining digitally observed features of known leukocytes using data cleaning, whereby outliers values are removed based on dry mass and area;

tuning the neural network as a function of at least one property of a precision-recall curve and F1 score representing leukocyte classifications of the known leukocytes generated by the neural network based on the digitally observed features;

configuring an imaging device with the trained and tuned neural network to generate data of observed leukocytes;

storing the data of observed leukocytes relating to observed leukocytes features; and

classifying the data of observed leukocytes into at least two classes by the trained and tuned neural network based on the observed leukocyte features.

2. The method of claim 1 , wherein the neural network is a residual neural network, visual geometry group, inception, AlexNet, fully connected neural network, convolutional neural network, or generative adversarial network (GAN).

3. The method of claim 1 , wherein the at least two leukocyte classes are selected from: granulocytes, lymphocytes, monocytes, and subclasses thereof.

4. The method of claim 3 , further comprising classifying lymphocytes into at least two subclasses selected from: natural killer cells, B cells, T cells, and subclasses thereof.

5. The method of claim 4 , further comprising classifying T cells into at least two subclasses selected from: CD8+ T cells, helper CD4+ T cells, regulatory CD4+ T cells, memory T cells, natural killer T cells, and gamma delta T cells.

6. The method of claim 3 , further comprising classifying granulocytes into at least two subclasses selected from: neutrophils, eosinophils, basophils, and mast cells.

7. The method of claim 3 , further comprising classifying monocytes into subclasses selected from: dendritic cells and macrophages.

8. The method of claim 1 , wherein the trained and tuned neural network is a two-stage neural network.

9. The method of claim 1 , wherein the observed features are derived from quantitative phase imaging of the observed leukocytes.

10. The method of claim 9 , wherein the quantitative phase imaging is diffraction phase microscopy.

11. The method of claim 1 , further comprising generating the precision-recall curve and the F1 score from the known leukocyte types or from the observed leukocytes.

12. The method of claim 1 , wherein at least one of the observed features is/are selected from: cell area, cell mass, cell morphology, cell granules, nuclear characteristics, and intracellular protein distributions.

13. The method of claim 1 , wherein the neural network comprises 2, 3, or 4 connected networks.

14. A leukocyte classifying system, the system comprising:

a quantitative phase microscope,

a processor,

a memory coupled to the processor, wherein the memory stores computer-readable instructions that, when executed by the process cause the processor to perform operations comprising:

obtaining digitally observed features of known leukocytes using data cleaning, whereby outliers values are removed based on dry mass and area;

identifying a plurality of leukocyte morphologies; and

classifying types of leukocyte by one or more neural network(s) based on the plurality of identified leukocyte morphologies.

15. The method of claim 14 , wherein the one or more neural network(s) include a residual neural network, visual geometry group, inception, AlexNet, fully connected neural network, convolutional neural network, or generative adversarial network (GAN).

16. The system of claim 14 , wherein the processor further performs operations comprising producing a digital interferogram associated with at least one of target leukocyte.

17. The system of claim 16 , wherein the processor further performs operations comprising extracting leukocyte features from the digital interferogram.

18. The system of claim 14 , wherein the processor classifies the types of leukocytes according to known classes defined based on known leukocyte morphologies.

19. The system of claim 14 , wherein at least one of the leukocyte morphologies is selected from: cell area, cell mass, cell morphology, nucleus characteristics, and intracellular protein distribution.

20. The system of claim 14 , wherein the processor is coupled with a classification database storing known leukocyte features correlated with known leukocyte types.

21. The system of claim 20 , wherein the known leukocyte types comprise granulocytes, lymphocytes, monocytes, and subclasses thereof.

22. The system of claim 21 , wherein the subclasses of lymphocytes are natural killer cells, B cells, T cells, and subclasses thereof.

23. The system of claim 22 , wherein the subclasses of T cells are CD8+ T cells, helper CD4+ T cells, regulatory CD4+ T cells, memory T cells, natural killer T cells, and/or gamma delta T cells.

24. The system of claim 21 , wherein the subclasses of granulocytes are neutrophils, eosinophils, basophils, and mast cells.

25. The system of claim 21 , wherein the subclasses of monocytes are dendritic cells and macrophages.

26. The system of claim 14 , wherein the processor classifies the types of leukocytes by a neural network trained according to known leukocyte phase maps.

27. The system of claim 14 , wherein the processor classifies the types of leukocytes using a neural network from donors whose leukocytes are not included in the training set.

28. The system of claim 14 , wherein the processor classifies the types of leukocytes using a neural network from donors whose leukocytes are included in the training set or from donors whose leukocytes are the only source of the training set.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 23, 2024
From: ZHOU, RENJIE; SHU, XIN; PANDEY, RISHIKESH
To: THE CHINESE UNIVERSITY OF HONG KONG
Reel/Frame 069665/0750 →
Continuity (2)
Provisional Application 63123111 · Dec 9, 2020
Related Publication 20220180515A1 · Jun 9, 2022
References Cited (81)
US 10176363B2 · El-Zehiry · 2019 [cited by examiner]
US 10203275B2 · Herzog · 2019 [cited by examiner]
US 20170212033A1 · Ozaki · 2017 [cited by examiner]
US 20180322327A1 · Smith · 2018 [cited by examiner]
US 20220172024A1 · Singh · 2022 [cited by examiner]
US 20230245651A1 · Wang · 2023 [cited by examiner]
US 20240038174A1 · Zhou · 2024 [cited by examiner]
Zhao et al., Automatic detection and classification of leukocytes using convolutional neural networks (Year: 2016). [cited by examiner]
Yu et al., Auutomatic classification of Leukocytes using Neural Network (Year: 2017). [cited by examiner]
Ridoy et al. An Automated Approach to White Blood Cell Classification Using a Lightweight Convolutional Neural Network (Year: 2020). [cited by examiner]
Su et al. A Neural Network Approach to White Blood Cell Classification (Year: 2014). [cited by examiner]
Bain, B.J., “Blood Cell Morphology in Health and Disease,” Dacie and Lewis Practical Haematology, 2017, pp. 61-92. [cited by applicant]
Thachil, J., et al., “Approach to the Diagnosis and Classification of Blood Cell Disorders,” Dacie and Lewis Practical Haematology, 2017, pp. 497-510. [cited by applicant]
Ciccullo, A., et al., “Neutrophil-to-lymphocyte ratio and clinical outcome in COVID-19: a report from the Italian front line,” International Journal of Antimicrobial Agents, 2020, 56:1-4. [cited by applicant]
Liu, J., et al., “Neutrophil-to-lymphocyte ratio predicts critical illness patients with 2019 coronavirus disease in the early stage,” Journal of Translational Medicine, 2020, 18:1-12. [cited by applicant]
Meintker, L., et al., “Comparison of Automated Differential Blood Cell Counts from Abbott Sapphire, Siemens Advia 120, Beckman Coulter DxH 800, and Sysmex XE-2100 in Normal and Pathologic Samples,” Am J Clin Pathol, 201… [cited by applicant]
Briggs, C., et al., “Basic Haematological Techniques,” Dacie and Lewis Practical Haematology, 2017, pp. 18-49. [cited by applicant]
Suzuki, Y., et al., “Label-free chemical imaging flow cytometry by high-speed multicolor stimulated Raman scattering,” PNAS, 2019, 116(32):15842-15848. [cited by applicant]
Blasi, T., et al., “Label-free cell cycle analysis for high-throughput imaging flow cytometry,” Nature Communications, 2016, 7:1-9. [cited by applicant]
Ojaghi, A., et al., “Label-free hematology analysis using deep-ultraviolet microscopy,” PNAS, 2020, 117(26):14779-14789. [cited by applicant]
Kobayashi, H., et al., “Label-free detection of cellular drug responses by high-throughput bright-field imaging and machine learning,” Scientific Reports, 2017, 7:1-9. [cited by applicant]
Heo, Y.J., et al., “Real-time Image Processing for Microscopy-based Label-free Imaging Flow Cytometry in a Microfluidic Chip,” Scientific Reports, 2017, 7:1-9. [cited by applicant]
Ramoji, A., et al., “Toward a Spectroscopic Hemogram: Raman Spectroscopic Differentiation of the Two Most Abundant Leukocytes from Peripheral Blood,” Analytical Chemistry, 2012, 84:5335-5342. [cited by applicant]
Mir, M., et al., “Quantitative Phase Imaging,” Progress in Optics, 2012, 57:133-217. [cited by applicant]
Park, Y., “Quantitative phase imaging in biomedicine,” Nature Photonics, 2018, 12:578-589. [cited by applicant]
Popescu, G., “Quantitative Phase Imaging of Cells and Tissues,” McGraw-Hill, 2011, pp. 1-384. [cited by applicant]
Popescu, G., et al., “Optical imaging of cell mass and growth dynamics,” Am J Physiol Cell Physiol, 2008, 295:C538-C544. [cited by applicant]
Mir, M., et al., “Optical measurement of cycle-dependent cell growth,” PNAS, 2011, 108(32):13124-13129. [cited by applicant]
Park, Y., et al., “Measurement of red blood cell mechanics during morphological changes,” PNAS, 2010, 107(15):6731-6736. [cited by applicant]
Popescu, G., et al., “Erythrocyte structure and dynamics quantified by Hilbert Phase microscopy,” Journal of Biomedical Optics, 2005, 10(6):1-3. [cited by applicant]
Li, Y., et al., “Quantitative investigation on morphology and intracellular transport dynamics of migrating cells,” Applied Optics, 2019, 58(34):G162-G168. [cited by applicant]
Lei, C., et al., “High-throughput imaging flow cytometry by optofluidic time-stretch microscopy,” Nature Protocols, 2018, 13:1603-1631, supplemental pp. 1-3. [cited by applicant]
Chen, C.L., et al., “Deep Learning in Label-free Cell Classification,” Scientific Reports, 2016, 6:1-16. [cited by applicant]
Zhang, J.K., et al., “Label-free colorectal cancer screening using deep learning and spatial light interference microscopy (SLIM),” APL Photonics, 2020, 5(040805):1-8. [cited by applicant]
Jo, Y., et al., “Holographic deep learning for rapid optical screening of anthrax spores,” Science Advances, 2017, 3:1-9, supplemental p. 1. [cited by applicant]
Jo, Y., et al., “Quantitative Phase Imaging and Artificial Intelligence: A Review,” IEEE Journal of Selected Topics in Quantum Electronics, 2019, 25(1):1-14. [cited by applicant]
Strbkova, L., et al., “Classification of Cells in Time-Lapse Quantitative Phase Image by Supervised Machine Learning,” Digital Holography and Three-Dimensional Imaging, 2019, pp. 1-2. [cited by applicant]
Lee, K.C.M., et al., “Quantitative Phase Imaging Flow Cytometry for Ultra-Large-Scale Single-Cell Biophysical Phenotyping,” Cytometry Part A, 2019, 95A:510-520. [cited by applicant]
Nissim, N., et al., “Real-Time Stain-Free Classification of Cancer Cells and Blood Cells Using Interferometric Phase Microscopy and Machine Learning,” Cytometry Part A, 2021, 99A:511-523. [cited by applicant]
Ozaki, Y., et al., “Label-free classification of cells based on supervised machine learning of subcellular structures,” PLOS One, 2019, 14(1):1-20. [cited by applicant]
O'Connor, T., et al., “Deep learning-based cell identification and disease diagnosis using spatio-temporal cellular dynamics in compact digital holographic microscopy,” Biomedical Optics Express, 2020, 11(8):4491-4508. [cited by applicant]
Kim, G., et al., “Learning-based screening of hematologic disorders using quantitative phase imaging of individual red blood cells,” Biosensors and Bioelectronics, 2019, 123:69-76. [cited by applicant]
Karandikar, S.H., et al., “Reagent-Free and Rapid Assessment of T Cell Activation State Using Diffraction Phase Microscopy and Deep Learning,” Analytical Chemistry, 2019, 91:3405-3411. [cited by applicant]
Yoon, J., et al., “Identification of non-activated lymphocytes using three-dimensional refractive index tomography and machine learning,” Scientific Reports, 2017, 7:1-10. [cited by applicant]
Li, Y., et al., “Accurate label-free 3-part leukocyte recognition with single cell lens-free imaging flow cytometry,” Computers in Biology and Medicine, 2018, 96:147-156. [cited by applicant]
Simonyan, K., et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv preprint arXiv:1409.1556 (2014). [cited by applicant]
Szagedy, C., et al., “Going Deeper with Convolutions,” Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9. [cited by applicant]
He, K., et al., “Deep Residual Learning for Image Recognition,” Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778. [cited by applicant]
He, K., et al., “Identity Mappings in Deep Residual Networks,” Proc. 2016 European Conference on Computer Vision, pp. 630-645. [cited by applicant]
Svensen, M., et al., “Pattern Recognition and Machine Learning,” Springer, 2006, pp. 1-101. [cited by applicant]
Grefenstette, J.J., “Genetic Algorithms and Machine Learning,” Proceedings of the sixth annual conference on Computational learning theory, 1993, pp. 1-2. [cited by applicant]
Rivenson, Y., et al., “PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning,” Light: Science & Applications, 2019, 8(23):1-11. [cited by applicant]
Lee, J., et al., “Deep-Learning-Based Label-Free Segmentation of Cell Nuclei in Time-Lapse Refractive Index Tomograms,” IEEE Access, 2019, 7:83449-83460. [cited by applicant]
Wang, H., et al., “Deep learning enables cross-modality super-resolution in fluorescence microscopy,” Nature Methods, 2019, 16:103-110, supplemental pp. 1-9. [cited by applicant]
Nassar, M., et al., “Label-Free Identification of White Blood Cells Using Machine Learning,” Cytometry Part A, 2019, 95A:836-842. [cited by applicant]
Habaza, M., et al., “Rapid 3D Refractive-Index Imaging of Live Cells in Suspension without Labeling Using Dielectrophoretic Cell Rotation,” Advanced Science, 2017, 4(1600205):1-9. [cited by applicant]
Niu, M., et al., “Portable quantitative phase microscope for material metrology and biological imaging,” Photonics Research, 2020, 8(7):1253-1259. [cited by applicant]
Wang, Z., et al., “Spatial light interference microscopy (SLIM)” Optics Express, 2011, 19(2):1-11. [cited by applicant]
Bhaduri, B., et al., “Diffraction phase microscopy: principles and applications in materials and life sciences,” Advances in Optics and Photonics, 2014, 6:57-119. [cited by applicant]
Jin, D., et al., “Large Population Cell Characterization Using Quantitative Phase Cytometer,” Cytometry Part A, 2017, 91A:450-459. [cited by applicant]
Hotelling, H., “Analysis of a Complex of Statistical Variables into Principal Components,” The Journal of Educational Psychology, 1933, pp. 417-441. [cited by applicant]
Van Der Maaten, L., et al., “Visualizing Data using t-SNE,” Journal of Machine Learning Research, 2008, 9:2579-2605. [cited by applicant]
Ioffe, S., et al., “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” Proceedings of the 32nd International Conference on Machine Learning, 2015, 37:1-9. [cited by applicant]
Nair, V., et al., “Rectified Linear Units Improve Restricted Boltzmann Machines,” Proceedings of the 27th International Conference on Machine Learning, 2010, pp. 1-8. [cited by applicant]
Bridle, J.S., “Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition,” Neurocomputing, 1990, 68:227-236. [cited by applicant]
Kingma, D.P., et al., “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980 (2014). [cited by applicant]
Robbins, H., et al., “A Stochastic Approximation Method,” The Annals of Mathematical Statistics, 1951, 22(3):400-407. [cited by applicant]
Manago, S., et al., “A reliable Raman-spectroscopy-based approach for diagnosis, classification and follow-up of B-cell acute lymphoblastic leukemia,” Scientific Reports, 2016, 6:1-13. [cited by applicant]
Wu, C.-H., et al., “Imaging Cytometry of Human Leukocytes with Third Harmonic Generation Microscopy,” Scientific Reports, 2016, 6:1-10. [cited by applicant]
Popescu, G., “Diffraction phase microscopy for quantifying cell structure and dynamics,” Optics Letters, 2006, 31(6):775-777. [cited by applicant]
Pandey, R., et al., “Integration of diffraction phase microscopy and Raman imaging for label-free morpho-molecular assessment of live cells,” Journal of Biophotonics, 2019, 12:1-8. [cited by applicant]
Barer, R., et al., “Interference Microscopy and Mass Determination,” Nature, 1952, 169:366-367. [cited by applicant]
Powers, D.M.W., “Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation,” Journal of Machine Learning Technologies, 2011, 2:37-63. [cited by applicant]
Saito, T., et al., “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets,” PLOS One, 2015, 10(3):1-21. [cited by applicant]
Jani, I.V., et al., “Effect of point-of-care CD4 cell count tests on retention of patients and rates of antiretroviral therapy initiation in primary health clinics: an observational cohort study,” Lancet, 2011, 378:1572… [cited by applicant]
Boyle, D.S., et al., “Emerging technologies for point-of-care CD4 T-lymphocyte counting,” Trends in Biotechnology, 2012, 30(1):45-54. [cited by applicant]
Olsson, J., et al., “Age-related change in peripheral blood T-lymphocyte subpopulations and cytomegalovirus infection in the very old: the Swedish longitudinal OCTO immune study,” Mechanisms of Ageing and Development, 2… [cited by applicant]
Brown, D.C., et al., “Ontogeny of T lymphocytes and intestinal morphological characteristics in neonatal pigs at different ages in the postnatal period,” J. Anim. Sci. 2006, 84:567-578. [cited by applicant]
Zheng, G., et al., “Wide-field, high-resolution Fourier ptychographic microscopy,” Nature Photonics, 2013, 7:739-745, supplemental p. 1. [cited by applicant]
Li, J., et al., “PURE-LET Image Deconvolution,” IEEE Transactions on Image Processing, 2018, 27(1):92-105. [cited by applicant]
Goldberg, D.E., et al., “Genetic Algorithms and Machine Learning,” Machine Learning, 1988, 3:95-99. [cited by applicant]