Data centric mislabel detection
A system or method for identifying mislabels in a training dataset. The labeled dataset includes multiple labeled samples, each of which is labeled with a ground-truth label. The labeled samples are divided into multiple training subset and hold-out test subset. For each training subset and corresponding hold-out test subset, a machine learning model is trained using the training subset, and the trained machine learning model is applied to the hold-out test subset to generate prediction labels for samples in the corresponding hold-out test subset. Each prediction label has a confidence score indicating a likelihood of the prediction label is correct. A pair of prediction label and ground truth label corresponding to a same sample is compared to determine whether there is a candidate mislabel. The system determines whether the candidate mislabel is a mislabel based in part on a confidence score of the prediction label.
1 . A computer-implemented method for identifying mislabels in a labeled dataset, the method comprising:
accessing a training dataset comprising a plurality of labeled samples, each of the plurality of labeled samples labeled with a ground-truth label;
dividing the plurality of labeled samples into a plurality of training subsets and hold-out test subsets;
for each of a training subset and a corresponding hold-out test subset in the plurality of training subsets and hold-out test subsets,
training a machine learning model using a corresponding training subset; and
applying the trained machine learning model to a corresponding hold-out test subset to generate prediction labels for samples in the corresponding hold-out test subset, wherein each prediction label has a confidence score indicating a likelihood that the prediction label is correct;
pairing prediction labels and ground truth labels corresponding to same samples;
comparing a pair of prediction label and ground truth label corresponding to a same sample to determine whether there is a candidate mislabel;
determining whether the candidate mislabel is a mislabel based in part on a confidence score of the prediction label; and
generating for display the determined mislabel.
2 . The computer-implemented method of claim 1 , wherein dividing the plurality of labeled samples into a plurality of pairs of training subset and hold-out test subset comprises:
dividing the plurality of labeled samples into N subsets, wherein N is a natural number that is greater than 1;
for each of N subsets,
selecting the subset as a hold-out test subset; and
selecting remaining (N−1) subsets as a training subset.
3 . The computer-implemented method of claim 1 , wherein the labeled dataset comprises a plurality of images, each of the plurality of images being labeled as a positive class or negative class, and each machine-learning model is trained as a classification model.
4 . The computer-implemented method of claim 1 , wherein the labeled dataset comprises a plurality of images, pixels of each of the plurality of images are labeled as positive or negative, and each machine-learning model is trained as segmentation models.
5 . The computer-implemented method of claim 1 , wherein the labeled dataset comprises a plurality of images, areas of each of the plurality of images are labeled as one of a plurality of objects, and each machine-learning model is trained as object detection models.
6 . The computer-implemented method of claim 5 , wherein pairing the prediction labels with the ground-truth labels comprises:
for a ground-truth label that corresponds to a first area of an image, identifying M prediction labels corresponding to M second areas of the image that overlap the first area, where M is a natural number; and
responsive to determining that M=1, pairing the prediction label with the ground-truth label.
7 . The computer-implemented method of claim 6 , wherein pairing the prediction labels with the ground-truth labels further comprises:
responsive to determining that M>1,
for each of the M prediction labels, determining a degree of overlap between the corresponding second area and the first area;
identifying a prediction label in the M prediction labels that correspond to a highest degree of overlap; and
pairing the prediction label with the ground-truth label.
8 . The computer-implemented method of claim 1 , wherein each machine-learning model is a classification model or a segmentation model,
a threshold is set based on an overall confidence level of all the prediction labels, and
when a confidence level of a prediction label corresponding to a candidate mislabel is greater than the threshold, the corresponding ground-truth label is determined as a mislabel.
9 . The computer-implemented method of claim 1 , wherein each machine-learning model is an object detection model configured to detect a plurality of classes of object,
wherein for each class of object, a class threshold is set based on an overall class confidence level, and
wherein when a confidence level of a prediction label of a class corresponding to a candidate mislabel is greater than the class threshold, the corresponding ground-truth label is determined as a mislabel.
10 . A computer program product, comprising a non-transitory computer-readable medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
access a training dataset comprising a plurality of labeled samples, each of the plurality of labeled samples labeled with a ground-truth label;
divide the plurality of labeled samples into a plurality of training subsets and hold-out test subsets;
for each of a training subset and a corresponding hold-out test subset in the plurality of training subsets and hold-out test subsets,
train a machine learning model using a corresponding training subset; and
apply the trained machine learning model to a corresponding hold-out test subset to generate prediction labels for samples in the corresponding hold-out test subset, wherein each prediction label has a confidence score indicating a likelihood that the prediction label is correct;
pair prediction labels and ground truth labels corresponding to same samples;
compare a pair of prediction label and ground truth label corresponding to a same sample to determine whether there is a candidate mislabel;
determine whether the candidate mislabel is a mislabel based in part on a confidence score of the prediction label; and
generate for display the determined mislabel.
11 . The computer program product of claim 10 , wherein dividing the plurality of labeled samples into a plurality of pairs of training subset and hold-out test subset comprises:
dividing the plurality of labeled samples into N subsets, where N is a natural number that is greater than 1;
for each of N subsets,
selecting the subset as a hold-out test subset; and
selecting remaining (N−1) subsets as a training subset.
12 . The computer program product of claim 10 , wherein the labeled dataset comprises a plurality of images, each of the plurality of images being labeled as a positive class or negative class, and each machine-learning model is trained as a classification model.
13 . The computer program product of claim 10 , wherein the labeled dataset comprises a plurality of images, pixels of each of the plurality of images are labeled as positive or negative, and each machine-learning model is trained as segmentation models.
14 . The computer program product of claim 10 , wherein the labeled dataset comprises a plurality of images, areas of each of the plurality of images are labeled as one of a plurality of objects, and each machine-learning model is trained as object detection models.
15 . The computer program product of claim 14 , wherein pairing the prediction labels with the ground-truth labels comprises:
for a ground-truth label that corresponds to a first area of an image, identifying M prediction labels corresponding to M second areas of the image that overlap the first area, where M is a natural number; and
responsive to determining that M=1, pairing the prediction label with the ground-truth label.
16 . The computer program product of claim 15 , wherein pairing the prediction labels with the ground-truth labels further comprises:
responsive to determining that M>1,
for each of the M prediction labels, determining a degree of overlap between the corresponding second area and the first area;
identifying a prediction label in the M prediction labels that correspond to a highest degree of overlap; and
pairing the prediction label with the ground-truth label.
17 . The computer program product of claim 10 , wherein each machine-learning model is a classification model or a segmentation model,
a threshold is set based on an overall confidence level of all the prediction labels, and
when a confidence level of a prediction label corresponding to a candidate mislabel is greater than the threshold, the corresponding ground-truth label is determined as a mislabel.
18 . The computer program product of claim 10 , wherein each machine-learning model is an object detection model configured to detect a plurality of classes of object,
for each class of object, a class threshold is set based on an overall class confidence level, and
when a confidence level of a prediction label of a class corresponding to a candidate mislabel is greater than the class threshold, the corresponding ground-truth label is determined as a mislabel.
19 . A computer system, comprising:
one or more processors; and
a non-transitory computer-readable medium, having instructions encoded thereon that, when executed by the one or more processors processor, cause the one or more processors to:
access a training dataset comprising a plurality of labeled samples, each of the plurality of labeled samples labeled with a ground-truth label;
divide the plurality of labeled samples into a plurality of training subsets and hold-out test subsets;
for each of a training subset and a corresponding hold-out test subset in the plurality of training subsets and hold-out test subsets,
train a machine learning model using a corresponding training subset; and
apply the trained machine learning model to a corresponding hold-out test subset to generate prediction labels for samples in the corresponding hold-out test subset, wherein each prediction label has a confidence score indicating a likelihood that the prediction label is correct;
pair prediction labels and ground truth labels corresponding to same samples;
compare a pair of prediction label and ground truth label corresponding to a same sample to determine whether there is a candidate mislabel;
determine whether the candidate mislabel is a mislabel based in part on a confidence score of the prediction label; and
generate for display the determined mislabel.
20 . The computer system of claim 19 , wherein dividing the plurality of labeled samples into a plurality of pairs of training subset and hold-out test subset comprises:
dividing the plurality of labeled samples into N subsets, where N is a natural number that is greater than 1;
for each of N subsets,
selecting the subset as a hold-out test subset; and
selecting a remaining (N−1) subsets as a training subset.