Automated stereology for determining tissue characteristics
Systems and methods for automated stereology are provided. In some embodiments, an active deep learning approach may be utilized to allow for a faster and more efficient training of a deep learning model for stereology analysis. In other embodiments, existing deep learning models for stereology analysis may be re-tuned to develop greater accuracy for a given data set of interest, either with or without an active deep learning approach. A method can include: capturing a data set including a stack of images of a three-dimensional (3D) object; determining whether an existing deep learning model is appropriate for use on the stack of images (or for re-tuning); performing pre-processing on the data set; performing a training of a deep learning model; applying the deep learning model to obtain a confidence score for each label of the data set; reviewing, by a user, at least some labels in the active set to verify whether the label displays sufficient agreement with an expected result, and moving only those that display sufficient agreement to a training set; and performing a stereology analysis using the trained deep learning model.
1 . A method of performing computerized stereology, the method comprising:
obtaining a data set comprising a test stack of images representing sections of a three-dimensional (3D) biological sample taken from a patient,
pre-processing the test stack of images using segmentation and focus algorithms;
dividing the data set into:
a training set comprising a first stack of images depicting annotated sections of the 3D biological sample, and
an active set comprising a second stack of images depicting unlabeled sections of the 3D biological sample;
accessing a library of existing deep learning models, and determining whether the library of existing deep learning models contains an existing deep learning model that was trained using labeled images having at least one metadata attribute common to at least one metadata attribute of the test stack of images, wherein the at least one metadata attribute comprises at least one of: a cell count analysis type for which the test stack of images was derived, a type of anatomy of the 3D biological sample depicted in the test stack of images, an expected type of cell depicted in the test stack of images, a type of method used to obtain the test stack of images, or a type of staining method used to prepare the 3D biological sample depicted in the test stack of images, by:
comparing data representative of the at least one metadata attribute of the test stack of images to data representative of corresponding metadata attributes of the labeled images used to train at least one existing deep learning model of the library of models, and
applying at least one existing deep learning model of the library of models to the test stack of images and determining whether the at least one existing deep learning model meets an accuracy threshold for the test stack of images;
when the metadata attributes of the labeled images used to train the library of existing deep learning models do not match the at least one metadata attribute of the test stack of images, perform an active deep learning process to generate a matching deep learning model, comprising:
(i) training a deep learning model using the training set, wherein the training of the deep learning model results in an initial trained deep learning model;
(ii) applying the initial trained deep learning model to the active set, and computing a confidence score for a predicted label corresponding to each image of the active set;
(iii) displaying to a user a subset of the active set, each image of the subset having a confidence score for each predicted label from a plurality of corresponding predicted labels, the confidence score having at least a predetermined threshold;
(iv) receiving validation information from the user relating to displayed images of the subset;
(v) moving each image of the active set for which the user agreed with the predicted label to the training set to create an expanded training set;
(vi) re-training the initial trained deep learning model using the expanded training set;
when the metadata attributes of the labeled images used to train the library of existing deep learning models do match the at least one metadata attribute of the test stack of images, performing steps (i)-(vi) on the matching existing deep learning model to retrain the model; and
applying the retrained deep learning model to the data set to perform a stereology analysis and display results to the user.
2 . The method according to claim 1 , wherein the segmentation algorithm applied to the data set comprises an adaptive segmentation algorithm (ASA).
3 . The method according to claim 2 , wherein the focus algorithm applied to the data set comprises an extended depth of field (EDF) algorithm.
4 . The method according to claim 1 , wherein the predetermined threshold is 90%.
5 . The method according to claim 1 , further comprising: (vii) storing a snapshot of the retrained model after step (vi).
6 . The method according to claim 5 , further comprising iterating steps (i)-(vii) for a predetermined number of iterations to create a snapshot ensemble.
7 . The method according to claim 6 , wherein the snapshot ensemble comprises snapshots of the retrained models saved every five epochs during predetermined number of iterations.
8 . The method according to claim 1 , wherein the deep learning model comprises a convolutional neural network (CNN).
9 . The method according to claim 1 , further comprising:
(vii) performing a post-processing step on the training set to remove noise, apply unbiased stereology rules, and apply a watershed algorithm.
10 . The method of claim 1 wherein the training set of step (i) of training the matching existing deep learning model comprises a set of the test stack of images plus a set of images from the training set.
11 . A system for performing computerized stereology, the system comprising:
an imager configured to capture a test stack of images of a three-dimensional (3D) biological sample taken from a patient, the test stack of images being a sequence of images of the 3D biological sample, and the imager being configured to capture the sequence of images of the 3D biological sample in increments having a step size along an axis of the 3D biological sample;
a processor in operable communication with the imager; and
a computer-readable medium in operable communication with the processor and having instructions stored thereon that, when executed, cause the processor to perform the following steps:
accessing a library of existing deep learning models, and determining whether the library of existing deep learning models contains an existing deep learning model that was trained using labeled imaged having at least one attributes common to metadata attributes-depicted in the test stack of images, wherein the at least one attributes comprises at least one of: a desired cell count analysis, a type of anatomy, an expected type of cell, a type of method used to of obtaining the test stack of images, or a type of staining method used in of staining a preparing the depicted 3D biological sample, by:
comparing data representative of the at least one attributes of the test stack of images to data representative of corresponding attributes of the metadata corresponding to the labeled images used to train at least one existing deep learning model of the library of models, and
applying at least one existing deep learning model of the library of models to the test stack of images and determining whether the at least one existing deep learning model meets an accuracy threshold for the test stack of images;
determining a confidence score having a threshold based on how closely the existing deep learning model from the plurality of existing deep learning models matches the test stack of images;
when the metadata's attributes corresponding to the labeled images used to train the library of existing deep learning models do not match the at least one attributes of the test stack of images perform an active deep learning process to generate a matching deep learning model, comprising:
i) perform pre-processing on the data set to convert each image of the test stack of images to gray and applying at least one algorithm to the data set to create a single image of the stack and to segment the single image into segments and provide respective labels for the segments;
ii) train a deep learning model to establish a ground truth using an adaptive segmentation algorithm and generate a snapshot ensemble of the deep learning model resulting in a trained deep learning model;
iii) apply the snapshot ensemble to an active set and compute a confidence score for each label of the respective labels for the segments, wherein the active set comprises a first stack of images corresponding to a pool of unlabeled sections of the 3D biological sample;
iv) display to a user all labels of the active set;
v) receive from the user an indication to select a subset of labels in the active set;
vi) move the subset of the labels to a training set comprising a second stack of images corresponding to annotated and segmented sections of the 3D biological sample; and
vii) perform a testing process on the training set including the subset using the trained deep learning model.
12 . The system according to claim 11 , wherein the at least one algorithm applied to the data set comprises an adaptive segmentation algorithm (ASA), extended depth of field (EDF) algorithm, or both.
13 . The system according to claim 11 , wherein the single image of the stack is an EDF image, and
wherein the labels are segmentation masks.
14 . The system according to claim 11 , wherein steps ii), iii), iv), v), and vii) are repeated a plurality of times before step v) is performed.
15 . The system according to claim 11 , wherein the deep learning model comprises a convolutional neural network (CNN).
16 . The system according to claim 11 , wherein the instructions, when executed, further cause the processor to perform the following step:
vi) perform a post-processing step on the train set to remove noise, apply unbiased stereology rules, and apply a watershed algorithm.
17 . The system according to claim 11 , wherein the at least one algorithm applied to the data set comprises an ASA and an EDF algorithm,
wherein the single image of the stack is an EDF image,
wherein the labels are segmentation masks,
wherein the snapshot ensemble comprises a plurality of snapshot models saved every five epochs during the training,
wherein steps ii), iii), iv), v), and vii) are repeated at least five times before step v) is performed,
wherein the deep learning model comprises a CNN, and
wherein the instructions, when executed, further cause the processor to perform the following step:
vi) performing, by the processor, a post-processing step on the train set to remove noise, apply unbiased stereology rules, and apply a watershed algorithm.