Systems and methods to process electronic images to categorize intra-slide specimen tissue type
Systems and methods are disclosed for identifying tissue specimen types present in digital whole slide images. In some aspects, tissue specimen types may be identified using unsupervised machine learning techniques for out-of-distribution detection. For example, a digital whole slide image of a tissue specimen and a recorded tissue specimen type for the digital whole slide image may be received. One or more feature vectors may be extracted from one or more foreground tiles of the digital whole slide image identified as including the tissue specimen, and a distribution learned by a machine learning system for the recorded tissue specimen type may be received. Using the distribution, a probability of the feature vectors corresponding to the recorded tissue specimen type may be computed and used as a basis for classifying the foreground tiles from which the feature vectors are extracted as an in-distribution foreground tile or an out-of-distribution foreground tile.
1 . A system for identifying one or more tissue specimen types by processing digital images, the system comprising:
a processor; and
a memory coupled to the processor and storing instructions that, when executed by the processor, cause the system to perform operations comprising:
receiving a digital image of a tissue specimen;
receiving a recorded tissue specimen type of the tissue specimen that was previously recorded and associated with the digital image;
providing the digital image as input to a trained machine learning system to receive, as output of the trained machine learning system, a prediction associated with one or more tissue specimen types present in the digital image;
comparing the prediction to the recorded tissue specimen type;
for at least a portion of the tissue specimen included in the digital image, determining a discrepancy between the prediction and the recorded tissue specimen type based on the comparing;
providing at least the portion of the tissue specimen included in the digital image corresponding to the discrepancy to a first image processing system specific to a tissue specimen type of the one or more tissue specimen types in the prediction; and
providing any remaining portion of the digital image of the tissue specimen to a second image processing system specific to the recorded tissue specimen type.
2 . The system of claim 1 , wherein the discrepancy includes one or more of:
a mismatch between the prediction and the recorded tissue specimen type, or
an additional tissue specimen type included in the prediction that is not included in the recorded tissue specimen type.
3 . The system of claim 1 , the operations further comprising:
generating and providing for display a notification indicating the discrepancy.
4 . The system of claim 3 , the operations further comprising:
providing the digital image for display, wherein the notification is displayed in association with at least the portion of the tissue specimen associated with the discrepancy within the displayed digital image.
5 . The system of claim 1 , further comprising:
based on the determined discrepancy, preventing at least the portion of the tissue specimen from being provided to the second image processing system.
6 . The system of claim 1 , wherein the trained machine learning system is an unsupervised machine learning system that learns a distribution for the recorded tissue specimen type, and using the distribution, the trained machine learning system is configured to output, as the prediction, a probability that the one or more tissue specimen types of the tissue specimen present in the digital image align with the recorded tissue specimen type.
7 . The system of claim 6 , wherein, based on the probability, at least the portion of the tissue specimen is classified as out-of-distribution to indicate the discrepancy, and any remaining portion of the tissue specimen is classified as in-distribution to indicate a match.
8 . The system of claim 1 , wherein the trained machine learning system is a supervised machine learning system that is trained to output, as the prediction, a predicted one or more tissue specimen types of the tissue specimen present in the digital image.
9 . The system of claim 8 , wherein the supervised machine learning system is trained using strong annotations by:
receiving, as training data, a plurality of training digital images of tissue specimen and strong annotations labeling a tissue specimen type included in each pixel of the plurality of training digital images; and
providing the training data as input to train the supervised machine learning system to predict one or more tissue specimen types of the tissue specimen.
10 . The system of claim 8 , wherein the supervised machine learning system is trained using multiple instance learning by:
receiving, as training data, a set of a plurality of training digital images of tissue specimen including one or more tissue specimen types and weak annotations labeling the one or more tissue specimen types as being present at an image level or tissue specimen level; and
providing the training data as input to train the supervised machine learning system to predict the one or more tissue specimen types of the tissue specimen, wherein the supervised machine learning system learns at least one common feature among the set that is representative of the one or more tissue specimen types.
11 . A method for identifying one or more tissue specimen types by processing digital images, the method comprising:
receiving a digital image of a tissue specimen;
receiving a recorded tissue specimen type of the tissue specimen that was previously recorded and associated with the digital image;
providing the digital image as input to a trained machine learning system to receive, as output of the trained machine learning system, a prediction associated with one or more tissue specimen types present in the digital image;
comparing the prediction to the recorded tissue specimen type;
for at least a portion of the tissue specimen included in the digital image, determining a discrepancy between the prediction and the recorded tissue specimen type based on the comparing;
providing at least the portion of the tissue specimen included in the digital image corresponding to the discrepancy to a first image processing system specific to a tissue specimen type of the one or more tissue specimen types in the prediction; and
providing any remaining portion of the digital image of the tissue specimen to a second image processing system specific to the recorded tissue specimen type.
12 . The method of claim 11 , wherein determining the discrepancy includes determining one or more of:
a mismatch between the prediction and the recorded tissue specimen type, or
an additional tissue specimen type included in the prediction that is not included in the recorded tissue specimen type.
13 . The method of claim 11 , further comprising:
generating and providing for display a notification indicating the discrepancy.
14 . The method of claim 13 , further comprising:
providing the digital image for display, wherein the notification is displayed in association with at least the portion of the tissue specimen associated with the discrepancy within the displayed digital image.
15 . The method of claim 11 , further comprising:
based on the determined discrepancy, preventing at least the portion of the tissue specimen from being provided to the second image processing system.
16 . The method of claim 11 , wherein, based on a type of the trained machine learning system, the prediction is one of:
a probability that the one or more tissue specimen types of the tissue specimen present in the digital image align with the recorded tissue specimen type; or
a predicted one or more tissue specimen types of the tissue specimen present in the digital image.
17 . A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations for identifying one or more tissue specimen types by processing digital images, the operations comprising:
receiving a digital image of a tissue specimen;
receiving a recorded tissue specimen type of the tissue specimen that was previously recorded and associated with the digital image;
providing the digital image as input to a trained machine learning system to receive, as output of the trained machine learning system, a prediction associated with one or more tissue specimen types present in the digital image;
comparing the prediction to the recorded tissue specimen type;
for at least a portion of the tissue specimen included in the digital image, determining a discrepancy between the prediction and the recorded tissue specimen type based on the comparing;
providing at least the portion of the tissue specimen included in the digital image corresponding to the discrepancy to a first image processing system specific to a tissue specimen type of the one or more tissue specimen types in the prediction; and
providing any remaining portion of the digital image of the tissue specimen to a second image processing system specific to the recorded tissue specimen type.