Guided workflow for deep learning error analysis
A model management system performs error analysis on results predicted by a machine learning model. The model management system identifies an incorrectly classified image outputted from a machine learning model and identifies using the Neural Template Matching (NTM) algorithm, an additional image correlated to the selected image. The system outputs correlated images based on a given image and a selection by a user through a user interface of a region of interest (ROI) of the given image. The region is defined by a bounding polygon input and the correlated images include features correlated to the features within the ROI. The system prompts a task associated with the additional image. The system receives a response that includes an indication that the additional image is incorrectly labeled and including a replacement label and instruct that the machine learning model be retrained using an updated training dataset that includes the replacement label.
1. A method comprising:
identifying an incorrectly classified image outputted from a machine learning model;
identifying, using a Neural Template Matching (NTM) model, an additional image that is correlated to the incorrectly classified image, wherein the NTM model outputs correlated images based on a given image and a selection by a user through a user interface of a region of interest (ROI) of the given image, the region defined by a bounding polygon input by the user, wherein the correlated images include features correlated to features within the ROI in the given image;
prompting, through the user interface, a task associated with the additional image;
receiving a response for the task from the user, through the user interface, the response including an indication that the additional image is incorrectly labeled and including a replacement label;
prompting, through the user interface, a second task to select an error type for further investigation from a presented group of error types, wherein the group of error types to present is selected based on a ranking of loss associated with each error type, the error type associated with a category of incorrectly classified images; and
instructing that the machine learning model be retrained using an updated training dataset that includes the replacement label.
2. The method of claim 1 , further comprising:
identifying a set of incorrectly classified images including the incorrectly classified image; and
classifying each of the set of incorrectly classified images into a category of a set of pre-determined categories based on an error type.
3. The method of claim 2 , wherein the task further comprises:
presenting the correlated image including the additional image to the user; and
presenting a message to the user to choose one or more images from the correlated images, wherein the one or more images are grouped incorrectly with the error type.
4. The method of claim 3 , wherein the one or more images chosen from the correlated images are chosen to be added to a training dataset; and wherein the method further comprises
instructing that the machine learning model be retrained with the training dataset.
5. The method of claim 2 , further comprising:
determining a score, for each incorrectly classified image in a group of incorrectly classified images of a same category; and
determining a ranking of the incorrectly classified images in the group based on the score.
6. The method of claim 5 , wherein the score is determined based on a loss function used in the machine learning model.
7. The method of claim 1 , further comprising:
responsive to detecting a selection of the error type, prompting, through the user interface, a plurality of additional tasks associated with the category of incorrectly classified images, each additional task being a concrete action for a user to perform for fixing the error type.
8. The method of claim 7 , wherein the plurality of additional tasks includes one or more of: presenting an image for the user to examine, presenting images to the user for relabeling, presenting a plurality of images for the user to choose from, or presenting two or more images for the user to compare.
9. The method of claim 7 , further comprising:
determining a root cause based on responses received to the plurality of additional tasks for the type of error; and
providing, through the user interface, an action that addresses the root cause of the type or error.
10. The method of claim 9 , wherein the action is one or more of: fixing image labels, updating a labeling convention, adding additional data for training, changing model hyperparameters, modifying data augmentation.
11. The method of claim 1 , wherein a type of error is one of: missed discoloration, missed crack, predicted crack instead of discoloration, predicted an extra crack, predicted an extra discoloration.
12. A non-transitory computer-readable storage medium storing executable computer instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the instructions comprising instructions to:
identify an incorrectly classified image outputted from a machine learning model;
identify, using a Neural Template Matching (NTM) model, an additional image that is correlated to the incorrectly classified image, wherein the NTM model outputs correlated images based on a given image and a selection by a user through a user interface of a region of interest (ROI) of the given image, the region defined by a bounding polygon input by the user, wherein the correlated images include features correlated to features within the ROI in the given image;
prompt, through the user interface, a task associated with the additional image;
receive a response for the task from the user, through the user interface, the response including an indication that the additional image is incorrectly labeled and including a replacement label;
prompt, through the user interface, a second task to select an error type for further investigation from a presented group of error types, wherein the group of error types to present is selected based on a ranking of loss associated with each error type, the error type associated with a category of incorrectly classified images; and
instructing that the machine learning model be retrained using an updated training dataset that includes the replacement label.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the instructions further comprising instructions to:
identify a set of incorrectly classified images including the incorrectly classified image; and
classify each of the set of incorrectly classified images into a category of a set of pre-determined categories based on an error type.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the task further comprises:
presenting the correlated image including the additional image to the user; and
presenting a message to the user to choose one or more images from the correlated images, wherein the one or more images are grouped incorrectly with the error type.
15. The non-transitory computer-readable medium of claim 14 , wherein the one or more images chosen from the correlated images are chosen to be added to a training dataset; and wherein the instructions further comprise instructions to:
instructing that the machine learning model be retrained with the training dataset.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions to:
determine a score, for each incorrectly classified image in a group of incorrectly classified images of a same category; and
determine a ranking of the incorrectly classified images in the group based on the score.
17. The non-transitory computer-readable medium of claim 16 , wherein the score is determined based on a loss function used in the machine learning model.
18. The non-transitory computer-readable medium of claim 12 , the instructions further comprising instructions to:
responsive to detecting a selection of the error type, prompt, through the user interface, a plurality of additional tasks associated with the category of incorrectly classified images, each additional task being a concrete action for a user to perform for fixing the error type.
19. The non-transitory computer-readable medium of claim 18 , wherein the plurality of additional tasks includes one or more of: presenting an image for the user to examine, presenting images to the user for relabeling, presenting a plurality of images for the user to choose from, or presenting two or more images for the user to compare.
20. The non-transitory computer-readable medium of claim 18 , the instructions further comprising instructions to:
determine a root cause based on responses received to the plurality of additional tasks for the type of error; and
provide, through the user interface, an action that addresses the root cause of the type or error.