IP Library › Granted Patent US 12,229,959
Granted Patent B2
US 12,229,959 · App. 17/971,295 · Granted Feb 18, 2025

Systems and methods for determining cell number count in automated stereology z-stack images

Inventors: Palak Pankajbhai Dave (Wesley Chapel, FL); Dmitry Goldgof (Lutz, FL); Lawrence O. Hall (Tampa, FL); Peter R. Mouton (Gulfport, FL)
Assignees: UNIVERSITY OF SOUTH FLORIDA; STEREOLOGY RESOURCE CENTER, INC.
G06T7/0014G06N3/08G06N20/00G06T5/20G06T5/70G06T7/11G06T7/174G06T7/97G06T17/205G06V20/695G06V20/698G06T2207/20081G06T2207/20084G06T2207/20152
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,229,959
App. No.
17/971,295
Granted
Feb 18, 2025
Kind
B2
Abstract

Systems and methods for automated stereology using deep learning are disclosed. The systems include an update in the form of a semi-automatic approach for ground truth preparation in 3D stacks of microscopy images (disector stacks) for generating more training data. The systems also present an exemplary disector-based MIMO framework where all the planes of a 3D disector stack are analyzed as opposed to a single focus-stacked image (EDF image) per stack. The MIMO approach avoids the costly computations of 3D deep learning-based methods by using the 3D context of cells in disector stacks; and prevents stereological bias in the previous EDF-based method due to counting profiles rather than cells and under-counting overlap-ping/occluded cells. Taken together, these improvements support the view that AI-based automatic deep learning methods can accelerate the efficiency of unbiased stereology cell counts without a loss of accuracy or precision as compared to conventional manual stereology.

Claims (65)

1. A method for performing computerized stereology, comprising:

obtaining a plurality of z-stack runtime images including one or more cells;

generating a plurality of z-stack grayscale images by converting the plurality of z-stack runtime images into grayscale;

applying the plurality of z-stack grayscale images to a trained deep-learning model, each z-stack grayscale image of the plurality of z-stack grayscale images corresponding to an input channel of the trained deep-learning model;

obtaining a plurality of outputs corresponding to the plurality of z-stack grayscale images from the trained deep-learning model, the plurality of outputs comprising information indicative of a plane of best focus for each cell of the one or more cells of the plurality of z-stack runtime images, wherein a first output N of the plurality of outputs is bidirectionally correlated with previous (N−1) and subsequent (N+1) outputs of the plurality of outputs; and

counting the one or more second cells in the plurality of outputs.

2. The method of claim 1 , wherein each Z-stack image of the plurality of Z-stack runtime images focuses on a different physical location on a z-axis, wherein the z-axis is perpendicular to 2D planes of the z-stack images.

3. The method of claim 2 , wherein the plurality of Z-stack runtime images is at a right angle to the z-axis.

4. The method of claim 1 , wherein the generating the plurality of z-stack grayscale images comprises:

removing a cresyl violet counterstain for a Nissl substance from the plurality of z-stack runtime images; and

generating the plurality of z-stack grayscale images by converting colors of the one or more second cells in the plurality of z-stack runtime images into grayscale.

5. The method of claim 1 , each output of the plurality of outputs comprises a segmentation map.

6. The method of claim 1 , further comprising:

identifying two overlapping cells in two consecutive outputs of the plurality of outputs;

combining the two overlapping cells into a single cell; and

assigning the single cell to an output of the two consecutive outputs, the output including a bigger cell of the two overlapping cells than another output of the two consecutive outputs.

7. The method of claim 1 , further comprising:

filling one or more hole of an output of the plurality of outputs to account for a solid cell body.

8. The method of claim 1 , further comprising:

discarding a cell overlapping another cell less than a predefined overlap with a minimum enclosing circle.

9. The method of claim 1 , wherein the deep learning model comprises a U-Net model.

10. The method of claim 1 , further comprising:

identifying one or more primary stains and one or more counter stains in the one or more second cells in the plurality of z-stack runtime images;

performing stain separation to remove the one or more counter-stain; and

filling one or more holes in the plurality of z-stack runtime images, the one or more holes caused from the stain separation.

11. The method of claim 1 , wherein each of the plurality of z-stack runtime images comprises a fluorescent image.

12. The method of claim 1 , wherein the training the deep learning model comprises:

generating one or more annotations on the one or more first cells in the plurality of z-stack training images to generate ground truth;

generating one or more binary image masks around the one or more first cells in the plurality of z-stack training images;

removing a subset of the one or more binary image masks, each mask of the subset omitting the one or more annotations; and

training the deep learning model based on the one or more annotations and the one or more binary image masks without the subset on the plurality of z-stack training images.

13. The method of claim 12 , wherein a mask of the one or more binary image masks corresponds to a cell of the one or more first cells, and wherein the mask of the one or more binary image mask corresponds to a z-stack training image of the plurality of the z-stack training images, the z-stack training image of the plurality of the z-stack training images being a best focal plane on the cell in the plurality of the z-stack training images.

14. A system for performing computerized stereology, comprising:

a processor;

a memory having stored thereon a set of instructions which, when executed by the processor, cause the processor to:

obtain a plurality of z-stack runtime images including one or more first cells;

generate a plurality of z-stack grayscale images by converting the plurality of z-stack runtime images into grayscale;

apply the plurality of z-stack grayscale images to a trained deep-learning model, each z-stack grayscale image of the plurality of z-stack grayscale images corresponding to an input channel of the trained deep-learning model;

obtain a plurality of outputs corresponding to the plurality of z-stack grayscale images from the trained deep-learning model, of the plurality of outputs comprising information indicative of a plane of best focus for each cell of the one or more cells of the plurality of z-stack runtime images, wherein a first output N of the plurality of outputs is bidirectionally correlated with previous (N−1) and subsequent (N+1) outputs of the plurality of outputs; and

count the one or more first cells in the plurality of outputs.

15. The system of claim 14 , wherein each Z-stack image of the plurality of Z-stack runtime images focuses on a different physical location on a z-axis.

16. The system of claim 15 , wherein the plurality of Z-stack runtime images is at a right angle to the z-axis.

17. The system of claim 14 , wherein to generate the plurality of z-stack grayscale images, the memory having stored thereon the set of instructions which cause the processor to:

remove a cresyl violet counterstain for a Nissl substance from the plurality of z-stack runtime images; and

generate the plurality of z-stack grayscale images by converting colors of the one or more second cells in the plurality of z-stack runtime images into grayscale.

18. The system of claim 14 , each output of the plurality of outputs comprises a segmentation map.

19. The system of claim 14 , wherein the memory having stored thereon the set of instructions which further cause the processor to:

identifying two overlapping cells in two consecutive outputs of the plurality of outputs;

combining the two overlapping cells into a single cell; and

assigning the single cell to an output of the two consecutive outputs, the output including a bigger cell of the two overlapping cells than another output of the two consecutive outputs.

20. The system of claim 14 , wherein the memory having stored thereon the set of instructions which further cause the processor to:

fill one or more holes of an output of the plurality of outputs to account for a solid cell body.

21. The system of claim 14 , wherein the memory having stored thereon the set of instructions which further cause the processor to:

discard a cell with less than a predefined overlap with a minimum enclosing circle.

22. The system of claim 14 , wherein the deep learning model comprises a U-Net model.

23. The system of claim 14 , wherein the memory having stored thereon the set of instructions which further cause the processor to:

identify one or more primary stains and one or more counter stains in the one or more second cells in the plurality of z-stack runtime images;

perform stain separation to remove the one or more counter-stains; and

fill one or more holes in the plurality of z-stack runtime images, the one or more holes caused from the stain separation.

24. The system of claim 14 , wherein each of the plurality of z-stack runtime images comprises a fluorescent image.

25. A method for performing computerized stereology, comprising:

obtaining a plurality of z-stack images including one or more cells of interest;

processing the plurality of z-stack images via a trained deep-learning model, each z-stack image of the plurality of z-stack images corresponding to an input channel of the trained deep-learning model;

obtaining a plurality of outputs corresponding to the plurality of z-stack images from the trained deep-learning model, the plurality of outputs comprising information indicative of a plane of best focus for each cell of the one or more cells of interest, wherein at least one output of the plurality of outputs represents the sole plane of best focus for a cell based on bidirectional comparison with neighboring outputs of the plurality of outputs; and

determining a count of the one or more cells of the z-stack of images based on the one or more cells' appearances in their respective planes of best focus in the plurality of outputs.

Continuity (6)
Continuation 17308592 · May 5, 2021
Continuation 16345392
Provisional Application 62420771 · Nov 11, 2016
Provisional Application 63263198 · Oct 28, 2021
Provisional Application 63357946 · Jul 1, 2022
Related Publication 20230127698A1 · Apr 27, 2023
References Cited (78)
US 8345946B2 · Ascenzi · 2013 [cited by applicant]
US 9739783B1 · Kumar et al. · 2017 [cited by applicant]
US 10473667B2 · Vu et al. · 2019 [cited by applicant]
US 20050267011A1 · Deisseroth et al. · 2005 [cited by applicant]
US 20090238457A1 · Rittscher et al. · 2009 [cited by applicant]
US 20090310833A1 · Ascenzi · 2009 [cited by applicant]
US 20100119119A1 · Rittscher et al. · 2010 [cited by applicant]
US 20100119127A1 · Bello et al. · 2010 [cited by applicant]
US 20120236120A1 · Kramer · 2012 [cited by examiner]
US 20120281883A1 · Hurley et al. · 2012 [cited by applicant]
US 20130157946A1 · Iqbal et al. · 2013 [cited by applicant]
US 20140131592A1 · Kishima · 2014 [cited by examiner]
US 20140347447A1 · Olesen · 2014 [cited by examiner]
US 20150260979A1 · Saito · 2015 [cited by examiner]
US 20150346191A1 · Aneja et al. · 2015 [cited by applicant]
US 20160131569A1 · Mai et al. · 2016 [cited by applicant]
US 20160196672A1 · Chertok et al. · 2016 [cited by applicant]
US 20160250355A1 · Macknik · 2016 [cited by applicant]
US 20180095450A1 · Lappas et al. · 2018 [cited by applicant]
US 20190130607A1 · Atchison · 2019 [cited by examiner]
Xing et al., Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review, IEEE Reviews in Biomedical Engineering, vol. 9, 2016, pp. 234-266. [cited by examiner]
Christiansen, E. M., Yang, S. J., Ando, D. M., Javaherian, A., Skibinski, G., Lipnick, S., Mount, E., O'Neil, A., Shah, K., Lee, A. K., et al., “In silico labeling: predicting fluorescent labels in unlabeled images,” Ce… [cited by applicant]
Dave, P., Goldgof, D., Hall, L.O., Kolinko, Y., Allen, K., Alahmari, S., Morera, H., Denham, G., Galvez, S., Becker, A., Albay, R., Mobley, W.C., and Mouton, P.R.. A novel automatic approach for total neuron counts usin… [cited by applicant]
Dave, P., Goldgof, D., Hall, L. O., Kolinko, Y., Allen, K., Alahmari, S., and Mouton, P. R., “A disector-based framework for the automatic optical fractionator,” Journal of Chemical Neuroanatomy, 102134 (2022). (pp. 1-1… [cited by applicant]
Gani, Md Osman, et al. “Multispectral object detection with deep learning.” Computational Intelligence in Communications and Business Analytics: Third International Conference, CICBA 2021, Santiniketan, India, Jan. 7-8,… [cited by applicant]
Štajduhar, A., Lepage, C., Judaš, M., Lončarić, S., & Evans, A. C. (Sep. 2018). 3d localization of neurons in bright-field histological images. In 2018 International Symposium ELMAR (pp. 75-78). IEEE. [cited by applicant]
Štajduhar, Andrija, et al. “Automatic detection of neurons in NeuN-stained histological images of human brain.” Physica A: Statistical Mechanics and its Applications 519 (2019): 237-246. [cited by applicant]
Mehnert, Andrew, et al. “A structural texture approach for characterising malignancy associated changes in pap smears based on mean-shift and the watershed transform.” 2014 22nd International Conference on Pattern Recog… [cited by applicant]
Benali et al., A Computerized Image Analysis System for Quantitative Analysis of Cells in Histological Brain Sections, Journal of Neuroscience Methods, 2003, 125:33-43. [cited by applicant]
Bonam et al., Toward Automated Quantification of Biological Microstructures Using Unbiased Stereology, Proceeding of SPIE, 2011, 7963:1-8. [cited by applicant]
Bradley et al., A One-Pass Extended Depth of Field Algorithm Based on the Over-Complete Discrete Wavelet Transform, 2004, pp. 279-284. [cited by applicant]
Chaudhury et al., A Novel Algorithm for Automated Counting of Stained Cells on Thick Tissue Sections, In 2012 25th IEEE International Symposium on Computer-Based Medical Systems (CBMS), pp. 1-6. [cited by applicant]
Chaudhury et al., An Ensemble Algorithm Framework for Automated Stereology of Cervical Cancer, In International Conference on Image Analysis and Processing, 2013, pp. 823-832. [cited by applicant]
Costa et al., Fast and Accurate Nonlinear Spectral Method for Image Recognition and Registration, Applied Physics Letters, 2006, 89(17):174102, pp. 1-3. [cited by applicant]
Elozory et al., Automatic Section Thickness Determination Using an Absolute Gradient Focus Function, Journal of Microscopy, 2012, 248(3):245-259. [cited by applicant]
Gardi et al., Automatic Sampling for Unbiased and Efficient Stereological Estimation Using the Proportionator in Biological Studies, Journal of Microscopy, 2008, 230(1):108-120. [cited by applicant]
Gundersen et al., Some New, Simple and Efficient Stereological Methods and Their Use in Pathological Research and Diagnosis, APMIS, 1988, 96:379-394. [cited by applicant]
Gundersen et al., The New Stereological Tools: Disector, Fractionator, Nucleator and Point Sampled Intercepts and Their Use in Pathological Research and Diagnosis, APMIS, 1988, 96:857-881. [cited by applicant]
Ho et al., NeurphologyJ: An Automatic Neuronal Morphology Quantification Method and Its Application in Pharmacological Discovery, BMC Bioinformatics, 2011, 12:230, 18 pages. [cited by applicant]
Inglis et al., Automated Identification of Neurons and Their Locations, Journal of Microscopy, 2008, 230(3):339-352. [cited by applicant]
Jensen et al., The Rotator, Journal of Microscopy, 1993, 170(1):35-44. [cited by applicant]
Kaplan et al., The Disector Counting Technique, NeuroQuantology, 2012, 10(1):44-53. [cited by applicant]
Lin et al., Hierarchical, Model-Bed Merging of Multiple Fragments for Improved Three-Dimensional Segmentation of Nuclei, Cytometry Part A, 2005, 63A:20-33. [cited by applicant]
Liu et al., High-Throughput, Automated Quantification of White Matter Neurons in Mild Malformation of Cortical Development in Epilepsy, Acta Neuropathologica Communications, 2014, 2(72):1-10. [cited by applicant]
Long et al., A New Preprocessing Approach for Cell Recognition, IEEE Transactions on Information Technology in Biomedicine, 2005, 9(3):407-412. [cited by applicant]
Long et al., Automatic Detection of Unstained Viable Cells in Bright Field Images Using a Support Vector Machine with an Improved Training Procedure, Computers in Biology and Medicine, 2006, 36:339-362. [cited by applicant]
Miller et al., Three Counting Methods Agree on Cell and Neuron Number in Chimpanzee Primary Visual Cortex, Frontiers in Neuroanatomy, 2014, 8(36)1-11. [cited by applicant]
Mouton et al., Design-Based Stereology and Video Densitometry for Assessment of Neurotoxicological Damage, Neurotoxicology, 2010, pp. 243-267. [cited by applicant]
Mouton, Applications of Unbiased Stereology to Neurodevelopmental Toxicology, Developmental Neurotoxicology Research, 2011, pp. 53-75. [cited by applicant]
Mouton et al., Automatic Stereology of Substantia Nigra Using a Novel Segmentation Framework Based on the Balloon Active Countour Model, Soc. Neurosci, 2015, vol. 735, 4 pages. [cited by applicant]
Mouton et al., Unbiased Stereology: A Concise Guide, The Johns Hopkins University Press, 2011, pp. 1-75. [cited by applicant]
Mouton, Quantitative Anatomy Using Design-Based Stereology, Handbook of Imaging in Biological Mechanics, 2014, pp. 217-228. [cited by applicant]
Mouton et al., Tg4510 Mice Provide an Effective Model for Testing Neuroprotective Therapies in Early-Stage Alzheimer's Disease, Soc. Neurosci, 2016, pp. 1-2. [cited by applicant]
Mouton et al., Unbiased Estimation of Cell Number Using the Automatic Optical Fractionator, Journal of Chemical Neuroanatomy, 2017, 80:A1-A8. [cited by applicant]
Nattkemper et al., A Neural Classifier Enabling High-Throughput Topological Analysis of Lymphocytes in Tissue Sections, IEEE Transactions on Information Technology in Biomedicine, 2001, 5(2):138-149. [cited by applicant]
Peng et al., Neuron Recognition by Parallel Potts Segmentation, PNAS, 2003, 100(7):3847-3852. [cited by applicant]
Phoulady et al., An Approach for Overlapping Cell Segmentation in Multi-Layer Cervical Cell Volumes, The Second Overlapping Cervical Cytology Image Segmentation Challenge—IEEE ISBI, 2015, 2 pages. [cited by applicant]
Phoulady et al., A New Approach to Detect and Segment Overlapping Cells in Multi-Layer Cervical Cell Volume Images, In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), 2016, pp. 201-204. [cited by applicant]
Phoulady et al., Experiments with Large Ensembles for Segmentation and Classification of Cervical Cancer Biopsy Images, IEEE International Conference on Systems, Man and Cybemetics, 2014, pp. 870-875. [cited by applicant]
Phoulady et al., Nucleus Segmentation in Histology Images with Hierarchical Multilevel Thresholding, Proceedings of SPIE, 2016, vol. 9791, pp. 1-6. [cited by applicant]
Ray et al., Tracking Leukocytes In Vivo with Shape and Size Constrained Active Contours, IEEE Transactions on Medical Imaging, 2002, 21(10):1222-1235. [cited by applicant]
Santacruz et al., Tau Suppression in a Neurodegenerative Mouse Model Improves Memory Function, Science, 2005, 309:476-481. [cited by applicant]
Savitzky et al., Smoothing and Differentiation of Data by Simplified Least Squares Procedures, Analytical Chemistry, 1964, 36(8):1627-1639. [cited by applicant]
Schmitz et al., Current Automated 3D Cell Detection Methods are not a Suitable Replacement for Manual Stereologic Cell Counting, Frontiers in Neuroanatomy, 2014, 8(27):1-13. [cited by applicant]
Sjostrom et al., Artificial Neural Network-Aided Image Analysis System for Cell Counting, Cytometry, 1999, 36:18-26. [cited by applicant]
Slater et al., A Machine Vision System for the Automated Classification and Counting of Neurons in 3-D Brain Tissue Samples, In Proceedings Third IEEE Workshop on Applications of Computer Vision, WACV '96, 1996, pp. 224… [cited by applicant]
Spires et al., Region-Specific Dissociation of Neuronal Loss and Neurofibrillary Pathology in a Mouse Model of Tauopathy, American Journal of Pathology, 2006, 168(5):1598-1607. [cited by applicant]
Sterio, The Unbiased Estimation of Number and Sizes of Arbitrary Particles Using the Disector, Journal of Microscopy, 1984, 134(2):127-136. [cited by applicant]
Tapias et al., Automated Imaging System for Fast Quantitation of Neurons, Cell Morphology and Neurite Morphometry In Vivo and In Vitro, Neurobiology of Disease, 2013, 54:156-168. [cited by applicant]
Valdecasas et al., On the Extended Depth of Focus Algorithms for Bright Field Microscopy, Micron, 2001, 32(6):559-569. [cited by applicant]
West et al., Unbiased Stereological Estimation of the Total Number of Neurons in the Subdivisions of the Rat Hippocampus Using the Optical Fractionator, The Anatomical Record, 1991, 231(4):482-497. [cited by applicant]
Wicksell, The Corpuscle Problem. A Mathematical Study of a Biometric Problem, Biometrika, 1925, 17(1/2):84-99. [cited by applicant]
Wikipedia, Convolutional Neural Network, Oct. 2016, 19 pages. [cited by applicant]
Xie et al., Deep Voting: A Robust Approach Toward Nucleus Localization in Microscopy Images, In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015, pp. 374-382. [cited by applicant]
Xing et al., Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review, IEEE Reviews in Biomedical Engineering, 2016, pp. 1-31. [cited by applicant]
Dong et al., Deep Learning for Automatic Cell Detection in Wide-Field Microscopy Zebrafish Images, In 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), 2015, pp. 772-776. [cited by applicant]
PCT International Search Report, PCT/US2017/061090, Mar. 20, 2018, 4 pages. [cited by applicant]
European Patent Office, Extended Search Report, U.S. Appl. No. 17/870,047, filed Jun. 30, 2021, 14 pages. [cited by applicant]