Semiautomatic machine learning model improvement and benchmarking

1. A method, comprising:

processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset;

receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing;

identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset;

determining a type of the first labeling error based on a second confusion matrix; and

modifying the training dataset based on the determined type of the first labeling error.

2. The method of claim 1 , wherein the first confusion matrix comprises a row-normalized confusion matrix, and wherein identifying the first labeling error in the training dataset comprises at least one of:

identifying at least two instances of a maximum value in a first row of the first confusion matrix, wherein the first row of the first confusion matrix corresponds to a first tag of at least one of the sets of tags; or

upon determining that one instance of a maximum value occurs in a first row of the first confusion matrix, determining that a position of the one instance of the maximum value is not on a diagonal of the first confusion matrix.

3. The method of claim 1 , wherein the second confusion matrix comprises a column-normalized matrix, and wherein determining the type of the first labeling error comprises:

computing an absolute value of a difference of two values in a first column of the second confusion matrix,

wherein when the computed absolute value does not exceed a threshold, the type of the first labeling error is a first type, and

wherein when the computed absolute value exceeds the threshold, the type of the first labeling error is a second type, different than the first type.

4. The method of claim 3 , wherein when the type of the first labeling error is determined to be the first type, modifying the training dataset comprises:

identifying a plurality of images determined to be mislabeled with a first tag of at least one of the sets of tags by the ML algorithm and the ML model; and

adding the identified plurality of images determined to be mislabeled with the first tag to the training dataset.

5. The method of claim 3 , wherein when the type of the first labeling error is determined to be the second type, modifying the training dataset comprises:

determining a second tag associated with the first column of the second confusion matrix; and

relabeling the first tag as the second tag in the training dataset.

6. The method of claim 3 , further comprising:

rebuilding the ML model based on the modified training dataset;

computing a score for a user providing the user input;

determining a message associated with the computed score; and

outputting an indication of the associated message to the user.

7. The method of claim 1 , wherein processing the plurality of images in the first dataset by the ML algorithm and the ML model comprises determining whether at least one object is depicted in a respective image, and wherein at least one image of the plurality of images in the first dataset and the training dataset comprise frames of video.

8. A computer program product, comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation comprising:

processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset;

receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing;

identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset;

determining a type of the first labeling error based on a second confusion matrix; and

modifying the training dataset based on the determined type of the first labeling error.

9. The computer program product of claim 8 , wherein the first confusion matrix comprises a row-normalized confusion matrix, and wherein identifying the first labeling error in the training dataset comprises at least one of:

identifying at least two instances of a maximum value in a first row of the first confusion matrix, wherein the first row of the first confusion matrix corresponds to a first tag of at least one of the sets of tags;

upon determining that one instance of a maximum value occurs in a first row of the first confusion matrix, determining that a position of the one instance of the maximum value is not on a diagonal of the first confusion matrix.

10. The computer program product of claim 8 , wherein the second confusion matrix comprises a column-normalized matrix, and wherein determining the type of the first labeling error comprises:

computing an absolute value of a difference of two values in a first column of the second confusion matrix,

wherein when the computed absolute value does not exceed a threshold, the type of the first labeling error is a first type, and

wherein when the computed absolute value exceeds the threshold, the type of the first labeling error is a second type, different than the first type.

11. The computer program product of claim 10 , wherein when the type of the first labeling error is determined to be the first type, modifying the training dataset comprises:

identifying a plurality of images determined to be mislabeled with a first tag of at least one of the sets of tags by the ML algorithm and the ML model; and

adding the identified plurality of images determined to be mislabeled with the first tag to the training dataset.

12. The computer program product of claim 10 , wherein when the type of the first labeling error is determined to be the second type, modifying the training dataset comprises:

determining a second tag associated with the first column of the second confusion matrix; and

relabeling the first tag as the second tag in the training dataset.

13. The computer program product of claim 10 , the operation further comprising: rebuilding the ML model based on the modified training dataset;

computing a score for a user providing the user input;

determining a message associated with the computed score; and

outputting an indication of the associated message to the user.

14. The computer program product of claim 8 , wherein processing the plurality of images in the first dataset by the ML algorithm and the ML model comprises determining whether at least one object is depicted in a respective image, and wherein at least one image of the plurality of images in the first dataset and the training dataset comprise frames of video.

15. A system, comprising:

one or more computer processors; and

a memory containing a program which when executed by the processors performs an operation comprising:

processing, by a machine learning (ML) algorithm and a ML model, a plurality of images in a first dataset, wherein the ML model was generated based on a plurality of images in a training dataset;

receiving user input reviewing a respective set of tags applied to each image in the first data set as a result of the processing;

identifying, based on a first confusion matrix generated based on the user input and the sets of tags applied to the images in the first data set, a first labeling error in the training dataset;

determining a type of the first labeling error based on a second confusion matrix; and

modifying the training dataset based on the determined type of the first labeling error.

16. The system of claim 15 , wherein the first confusion matrix comprises a row-normalized confusion matrix, and wherein identifying the first labeling error in the training dataset comprises at least one of:

identifying at least two instances of a maximum value in a first row of the first confusion matrix, wherein the first row of the first confusion matrix corresponds to a first tag of at least one of the sets of tags;

upon determining that one instance of a maximum value occurs in a first row of the first confusion matrix, determining that a position of the one instance of the maximum value is not on a diagonal of the first confusion matrix.

17. The system of claim 15 , wherein the second confusion matrix comprises a column-normalized matrix, and wherein determining the type of the first labeling error comprises:

computing an absolute value of a difference of two values in a first column of the second confusion matrix,

wherein when the computed absolute value does not exceed a threshold, the type of the first labeling error is a first type, and

wherein when the computed absolute value exceeds the threshold, the type of the first labeling error is a second type, different than the first type.

18. The system of claim 17 , wherein when the type of the first labeling error is determined to be the first type, modifying the training dataset comprises:

identifying a plurality of images determined to be mislabeled with a first tag of at least one of the sets of tags by the ML algorithm and the ML model; and

adding the identified plurality of images determined to be mislabeled with the first tag to the training dataset.

19. The system of claim 17 , wherein when the type of the first labeling error is determined to be the second type, modifying the training dataset comprises:

determining a second tag associated with the first column of the second confusion matrix; and

relabeling the first tag as the second tag in the training dataset.

20. The system of claim 17 , wherein processing the plurality of images in the first dataset by the ML algorithm and the ML model comprises determining whether at least one object is depicted in a respective image, and wherein at least one image of the plurality of images in the first dataset and the training dataset comprise frames of video, the operation further comprising:

rebuilding the ML model based on the modified training dataset;

computing a score for a user providing the user input;

determining a message associated with the computed score; and

outputting an indication of the associated message to the user.