Characterizing a classification difficulty associated with a target data item

1 . A method for identifying potentially mislabeled data samples, the method comprising: receiving a dataset comprising a plurality of data items, wherein at least a portion of the plurality of data items is each associated with a corresponding class label from labels of a plurality of classes;

for each class of the plurality of classes, training a separate class-conditional reconstructor on one or more of the data items associated with that class; and

for a target data item in the dataset having a target class label among the labels of the plurality of classes:

calculating a first reconstruction error using a class-conditional reconstructor trained for the target class label;

calculating a second reconstruction error using a class-conditional reconstructor trained for a class other than the target class label;

determining a reconstruction error ratio of the first reconstruction error to the second reconstruction error;

using the reconstruction error ratio, characterizing a classification associated with the target data item;

determining that the target data item is potentially mislabeled in response to determining that the reconstruction error ratio exceeds a certain threshold;

assigning the target data item to one of a plurality of confidence categories based on the reconstruction error ratio; and

automatically performing a dataset modification operation based on the assigned confidence category, including by excluding the target data item from a training dataset used to train a downstream machine learning classifier.

2 . The method of claim 1 , wherein the data items comprise image data.

3 . The method of claim 1 , wherein each class-conditional reconstructor is a shallow autoencoder.

4 . The method of claim 1 , wherein the second reconstruction error is a minimum reconstruction error for a group of reconstructions calculated using a class-conditional reconstructor trained for a class other than the target class label.

5 . The method of claim 1 , wherein the classification associated with the target data item is a classification difficulty for the target data item.

6 . The method of claim 5 , wherein the classification difficulty is determined based on how close the first reconstruction error is to the second reconstruction error.

7 . The method of claim 1 , wherein the classification associated with the target data item is based on whether the target data item is potentially mislabeled.

8 . The method of claim 1 , further comprising calculating a classification difficulty for the entire dataset.

9 . The method of claim 8 , wherein the classification difficulty for the entire dataset is associated with an average of the reconstruction error ratio across all data items in the dataset.

10 . The method of claim 1 , further comprising providing the classification associated with the target data item as an output.

11 . The method of claim 1 , further comprising assigning the target data item to one of a plurality of confidence categories based on the ratio of reconstruction errors.

12 . The method of claim 11 , wherein assigning the target data item to one of the plurality of confidence categories includes comparing the ratio of reconstruction errors to one or more predetermined thresholds.

13 . The method of claim 12 , wherein the confidence categories comprise a first category indicating high confidence in label correctness, a second category indicating low confidence in label correctness, and a third category indicating uncertainty.

14 . The method of claim 11 , further comprising providing the category assignment of one or more data items to a user.

15 . The method of claim 1 , further comprising transforming the data items into feature vectors by feeding them through a pretrained artificial intelligence model.

16 . The method of claim 1 , further comprising computing a probability that a target data item is mislabeled based on the determined ratio of reconstruction errors.

17 . The method of claim 1 , further comprising providing a probability that one or more of the data items are mislabeled to a user.

18 . A system for identifying potentially mislabeled data samples, the system comprising:

a processor configured to:

receive a dataset comprising a plurality of data items, wherein at least a portion of the plurality of data items is each associated with a corresponding class label from labels of a plurality of classes;

for each class of the plurality of classes, train a separate class-conditional reconstructor on one or more of the data items associated with that class; and

for a target data item in the dataset having a target class label among the labels of the plurality of classes:

calculate a first reconstruction error using a class-conditional reconstructor trained for the target class label;

calculate a second reconstruction error using a class-conditional reconstructor trained for a class other than the target class label;

determine a reconstruction error ratio of the first reconstruction error to the second reconstruction error;

use the reconstruction error ratio, characterize a classification associated with the target data item;

determine that the target data item is potentially mislabeled in response to determining that the reconstruction error ratio exceeds a certain threshold;

assign the target data item to one of a plurality of confidence categories based on the reconstruction error ratio; and

automatically perform a dataset modification operation based on the assigned confidence category, including by excluding the target data item from a training dataset used to train a downstream machine learning classifier; and

a memory coupled to the processor and configured to provide the processor with instructions.

19 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

receiving a dataset comprising a plurality of data items, wherein at least a portion of the plurality of data items is each associated with a corresponding class label from labels of a plurality of classes;

for each class of the plurality of classes, training a separate class-conditional reconstructor on one or more of the data items associated with that class; and

for a target data item in the dataset having a target class label among the labels of the plurality of classes:

calculating a first reconstruction error using a class-conditional reconstructor trained for the target class label;

calculating a second reconstruction error using a class-conditional reconstructor trained for a class other than the target class label;

determining a reconstruction error ratio of the first reconstruction error to the second reconstruction error;

using the reconstruction error ratio, characterizing a classification associated with the target data item;

determining that the target data item is potentially mislabeled in response to determining that the reconstruction error ratio exceeds a certain threshold;

assigning the target data item to one of a plurality of confidence categories based on the reconstruction error ratio; and

automatically performing a dataset modification operation based on the assigned confidence category, including by excluding the target data item from a training dataset used to train a downstream machine learning classifier.