Techniques for improving data efficiency in image-based machine learning
In example embodiments, a dataset labeling software process combines diversity selection and with one or more uncertainty selection techniques in a multi-stage process that produces a labeled training dataset for training an image-based ML model. In an initial stage, diversity selection alone may be used to seed a labeled training dataset. In a subsequent stage, diversity selection may be combined with a first procedure for computing uncertainty (e.g., uncertainty selection based on ensemble learning) to build the labeled training dataset until a first stopping condition is met. Optionally, in a still further stage, diversity selection may be combined with a second, different procedure for computing uncertainty (e.g., uncertainty selection based on Monte Carlo (MC) dropout) to further build the labeled training dataset until a second stopping condition is met.
1 . A method for training an image-based machine learning (ML) model, comprising:
in an initial stage,
computing, by a software process executing on one or more computing devices, a dissimilarity between each image of an initial set of images available for training, and
selecting, by the software process, a first sub-set of images from the initial set of images based on the dissimilarity from each other, labeling the first sub-set of images, and adding the labeled first sub-set of images to a labeled training dataset;
training one or more image-based ML models on the labeled training dataset;
in a subsequent stage,
computing, by the software process, an uncertainty of the one or more image-based ML models for remaining images of the initial set of images, and
selecting, by the software process, a second sub-set of images from the remaining images of the initial set of images based on a combination of the dissimilarity from each other and the uncertainty of the one or more image-based ML models, labeling the second sub-set of images, and adding the second sub-set of images to the labeled training dataset;
repeating the training the one or more image-based ML models, and the computing the uncertainty of the one or more image-based ML models, and the selecting the second sub-set of images of the subsequent stage until a stopping condition is met to build the labeled training dataset; and
using the labeled training dataset to train the image-based ML model.
2 . The method of claim 1 , wherein the one or more image-based ML models are an ensemble of image-based ML models, and the uncertainty of the one or more image-based ML models for each remaining image is a disagreement score for each image that measures disagreement between predictions of each of the image-based ML models of the ensemble.
3 . The method of claim 2 , wherein the stopping condition is that an aggregate of the disagreement score for each image is below a disagreement threshold.
4 . The method of claim 1 , further comprising:
further building the labeled training dataset by repeating the training the one or more image-based ML models, the computing the uncertainty of the one or more image-based ML models, and the selecting the second sub-set of images in a still further stage using a different procedure for computing uncertainty.
5 . The method of claim 1 , further comprising:
training a single image-based ML model on the labeled training dataset;
in a still further stage,
computing, by the software process, an uncertainty of the single image-based ML model for the remaining images of the initial set of images using Monte-Carlo dropout, and
selecting, by the software process, a third sub-set of images from the remaining images of the initial set of images based on a combination of the dissimilarity from each other and the uncertainty of the single image-based ML model, labeling the third sub-set of images, and adding the third sub-set of images to the labeled training dataset; and
repeating the training the ML model, and the computing the uncertainty of the single image-based ML model, and the selecting the third sub-set of images of the still further stage until a further stopping condition is met.
6 . The method of claim 5 , wherein the uncertainty of the single image-based ML model is a disagreement score that measures disagreement between predictions of the single image-based ML model with different weights set to zero.
7 . The method of claim 5 , wherein the further stopping condition is that a predicted accuracy improvement of adding additional labeled images to the labeled training dataset is below an improvement threshold.
8 . The method of claim 1 , wherein the computing the dissimilarity further comprises:
encoding each image of the initial set of images into encodings that represent features;
calculating similarity measures between each of the encodings;
organizing the calculated similarity measures between each of the encodings into a similarity matrix that quantifies how similar each of the images are to each other; and
calculating a diversity metric for each image of the initial set of images based on a line in the similarity matrix that corresponds to the image.
9 . The method of claim 8 , wherein the selecting the first sub-set of images selects images having greatest dissimilarity based on the diversity metric of each image.
10 . The method of claim 1 , further comprising:
using the trained image-based ML model to form a prediction for a new image.
11 . The method of claim 10 , further comprising:
computing, by the software process, a dissimilarity between the new image and the labeled training dataset;
computing, by the software process, an uncertainty of the prediction for the new image;
flagging the new image for review based on the dissimilarity between the new image and the labeled training dataset, the uncertainty of prediction for the new image or a combination of the dissimilarity between the new image and the labeled training dataset and the uncertainty of prediction for the new image exceeding a threshold; and
outputting an indication that the prediction for the flagged new image is potentially inaccurate.
12 . A non-transitory computing device readable medium having instructions stored thereon, the instructions when executed by one or more computing devices operable to:
in an initial stage,
compute a dissimilarity between each image of an initial set of images in available for training, and
select a first sub-set of images from the initial set of images based on the dissimilarity from each other, label the first sub-set of images, and add the labeled first sub-set of images to a labeled training dataset;
in a subsequent stage,
compute an uncertainty of one or more image-based machine learning (ML) models for each image of the initial set of images, and
select a second sub-set of images from the initial set of images based on a combination of the dissimilarity from each other and the uncertainty, label the second sub-set of images, and add the sub-set of images to the labeled training dataset;
repeat the computation of the uncertainty the selection of the sub-set of images of the subsequent stage until a stopping condition is met to build the labeled training dataset; and
use the labeled training dataset to train an image-based ML model.
13 . The non-transitory electronic-device readable medium of claim 12 , wherein the one or more image-based ML models are an ensemble of image-based ML models, and the instructions operable to compute the uncertainty further comprise instructions operable to:
compute a disagreement score for each image that measures disagreement between predictions of each of the image-based ML models of the ensemble.
14 . The non-transitory electronic-device readable medium of claim 12 , wherein the one or more image-based ML models are a single image-based ML model and the instructions operable to compute the uncertainty further comprise instructions operable to:
compute a disagreement score that measures disagreement between predictions of the single image-based ML model with different weights set to zero by Monte Carlo dropout.
15 . The non-transitory electronic-device readable medium of claim 14 , wherein the instructions operable to computing the dissimilarity comprise instructions operable to
encode each image of the initial set of images into encodings that represent features;
calculate similarity measures between each of the encodings;
organize the calculated similarity measures between each of the encodings into a similarity matrix that quantifies how similar each of the images are to each other; and
calculate a diversity metric for each image of the initial set of images based on a line in the similarity matrix that corresponds to the image.
16 . The non-transitory electronic-device readable medium of claim 12 , wherein the stopping condition is that a predicted accuracy improvement of adding additional labeled images to the labeled training dataset is below an improvement threshold.
17 . A computing device configured to train an image-based machine learning (ML) model, comprising:
one or more processors; and
a memory configured to store software for execution on the one or more processors, the software including a data labeling software process that is configured to:
in an initial stage,
compute a dissimilarity between each image of an initial set of images available for training, and
select a first sub-set of images from the initial set of images based on the dissimilarity from each other;
provide the first sub-set of images for training one or more image-based ML models;
in a subsequent stage,
compute an uncertainty of the one or more image-based ML models for remaining images of the initial set of images, and
select a second sub-set of images from the remaining images of the initial set of images based on a combination of the dissimilarity from each other and the uncertainty of the one or more image-based ML models;
provide the second sub-set of images for training the one or more image-based ML models; and
repeat the computation of the uncertainty of the one or more image-based ML models, and the selection of the second sub-set of images of the subsequent stage, and the provision of the second sub-set of images for training until a stopping condition is met.
18 . The computing device of claim 17 , wherein the one or more image-based ML models are an ensemble of image-based ML models, the uncertainty of the one or more image-based ML models for each remaining image is a disagreement score for each image that measures disagreement between predictions of each of the image-based ML models of the ensemble, and the stopping condition is that an aggregate of the disagreement score for each image is below a disagreement threshold.
19 . The computing device of claim 17 , wherein the data labeling software process is further configured to:
encode each image of the initial set of images into encodings that represent features;
calculate similarity measures between each of the encodings;
organize the calculated similarity measures between each of the encodings into a similarity matrix that quantifies how similar each of the images are to each other; and
calculate a diversity metric for each image of the initial set of images based on a line in the similarity matrix that corresponds to the image,
wherein the first sub-set of images is selected as images having greatest dissimilarity based on the diversity metric of each image.
20 . The computing device of claim 17 , wherein the software further includes a trained image-based ML model that has been trained using the first sub-set of images and the second sub-set of images, and the trained image-based ML model is configured to:
form a prediction for a new image.