SYSTEMS AND METHODS FOR PROCESSING ELECTRONIC IMAGES USING UNCERTAINTY ESTIMATION
A method for processing electronic images using uncertainty estimation may be used to determine whether to use an artificial intelligence (AI) assisted prediction. The method may include receiving one or more electronic images associated with a pathology specimen and providing the one or more electronic images to a machine learning model. The machine learning model may perform operations including determining a certainty level corresponding to a certainty that a predetermined AI system will provide an accurate prediction, determining whether the certainty level equals or exceeds a predetermined confidence threshold, and, upon determining that the certainty level does not equal or exceed a predetermined confidence threshold, determining to not use the predetermined AI system.
1 . A computer-implemented method of determining whether to use an artificial intelligence (AI)-assisted prediction, the method comprising:
receiving, by one or more processors, one or more electronic slide images associated with a pathology specimen;
determining, by a trained machine learning model, a certainty level associated with a predetermined AI system based on classifying the one or more electronic slide images;
determining, by the trained machine learning model, that the certainty level does not equal or exceed a predetermined confidence threshold; and
outputting, by the trained machine learning model, a determination not to use the predetermined AI system for the AI-assisted prediction for the pathology specimen based on determining that the certainty level does not equal or exceed the predetermined confidence threshold.
2 . The computer-implemented method of claim 1 , wherein determining the certainty level further comprises identifying a subpopulation of the pathology specimen.
3 . The computer-implemented method of claim 1 , wherein determining that the certainty level does not equal or exceed the predetermined confidence threshold further comprises:
determining a classified label, wherein the classified label indicates that the pathology specimen is outside of one or more predetermined subpopulations.
4 . The computer-implemented method of claim 3 , wherein determining the classified label further comprises:
partitioning the one or more electronic slide images into a plurality of foreground tiles;
extracting a vector of features from each foreground tile; and
running each extracted vector through the trained classifier to determine the classified label.
5 . The computer-implemented method of claim 1 , wherein determining that the certainty level does not equal or exceed the predetermined confidence threshold further comprises:
detecting one or more features in the one or more electronic slide images; and
determining a consistency of the detected one or more features.
6 . The computer-implemented method of claim 1 , wherein the predetermined confidence threshold is based on a second certainty level associated with an alternative method.
7 . The computer-implemented method of claim 1 , further comprising:
outputting a recommendation for an alternative method for determining a prediction for the pathology specimen.
8 . A system for determining whether to use an artificial intelligence (AI)-assisted prediction, the system comprising:
a memory storing instructions; and
one or more processors configured to execute the instructions to perform operations comprising:
receiving, by the one or more processors, one or more electronic slide images associated with a pathology specimen;
determining, by a trained machine learning model, a certainty level associated with a predetermined AI system based on classifying the one or more electronic slide images;
determining, by the trained machine learning model, that the certainty level does not equal or exceed a predetermined confidence threshold; and
outputting, by the trained machine learning model, a determination not to use the predetermined AI system for the AI-assisted prediction for the pathology specimen based on determining that the certainty level does not equal or exceed the predetermined confidence threshold.
9 . The system of claim 8 , wherein determining the certainty level further comprises identifying a subpopulation of the pathology specimen.
10 . The system of claim 8 , wherein determining that the certainty level does not equal or exceed the predetermined confidence threshold further comprises:
determining a classified label, wherein the classified label indicates that the pathology specimen is outside of one or more predetermined subpopulations.
11 . The system of claim 10 , wherein determining the classified label further comprises:
partitioning the one or more electronic slide images into a plurality of foreground tiles;
extracting a vector of features from each foreground tile; and
running each extracted vector through the trained classifier to determine the classified label.
12 . The system of claim 8 , wherein determining that the certainty level does not equal or exceed the predetermined confidence threshold further comprises:
detecting one or more features in the one or more electronic slide images; and
determining a consistency of the detected one or more features.
13 . The system of claim 8 , wherein the predetermined confidence threshold is based on a second certainty level associated with an alternative method.
14 . The system of claim 8 , the operations further comprising:
outputting a recommendation for an alternative method for determining a prediction for the pathology specimen.
15 . A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, performs a method for determining whether to use an artificial intelligence (AI)-assisted prediction, the method comprising:
receiving, by the one or more processors, one or more electronic slide images associated with a pathology specimen;
determining, by a trained machine learning model, a certainty level associated with a predetermined AI system based on classifying the one or more electronic slide images;
determining, by the trained machine learning model, that the certainty level does not equal or exceed a predetermined confidence threshold; and
outputting, by the trained machine learning model, a determination not to use the predetermined AI system for the AI-assisted prediction for the pathology specimen based on determining that the certainty level does not equal or exceed the predetermined confidence threshold.
16 . The non-transitory computer-readable medium of claim 15 , wherein determining the certainty level further comprises identifying a subpopulation of the pathology specimen.
17 . The non-transitory computer-readable medium of claim 15 , wherein determining that the certainty level does not equal or exceed the predetermined confidence threshold further comprises:
determining a classified label, wherein the classified label indicates that the pathology specimen is outside of one or more predetermined subpopulations.
18 . The non-transitory computer-readable medium of claim 17 , wherein determining the classified label further comprises:
partitioning the one or more electronic slide images into a plurality of foreground tiles;
extracting a vector of features from each foreground tile; and
running each extracted vector through the trained classifier to determine the classified label.
19 . The non-transitory computer-readable medium of claim 15 , wherein determining that the certainty level does not equal or exceed the predetermined confidence threshold further comprises:
detecting one or more features in the one or more electronic slide images; and
determining a consistency of the detected one or more features.
20 . The non-transitory computer-readable medium of claim 15 , wherein the predetermined confidence threshold is based on a second certainty level associated with an alternative method.