Domain-specific human-model collaborative annotation tool
View Patent ↗A human-model collaborative annotation system for training human annotators includes a database that stores images previously annotated by an expert human annotator and/or a machine learning annotator, a display that displays images selected from the database, an annotation system that enables human annotators to annotate images presented on the display, and an annotation training system. The annotation training system selects an image sample from the database for annotation by a human annotator, receives one or more proposed annotations from the annotation system, compares the human annotator's one or more proposed annotations to previous annotations of the image sample by the expert human annotator or machine learning annotator, presents attention maps on the display to draw the human annotator's attention to any annotation errors identified by the comparing, and selects a next training image sample from the database based on any errors identified in the comparing step.
1. A method for training human labelers to label images, the method comprising:
receiving, by one or more processors, a set of domain-specific images;
presenting, by the one or more processors, an image sample from the set of domain-specific images on a display to a human labeler for labeling via a graphical user interface, wherein the image sample has been previously labeled by at least one of an expert human and labeler or a machine learning model trained on expert-labeled images;
receiving, by one or more processors, one or more proposed labels from the human labeler through the graphical user interface;
comparing, by the one or more processors, the one or more proposed labels to previous labels of the image sample by the expert human labeler or machine learning model;
generating, by the one or more processors, computer-generated attention maps comprising a visual overlay superimposed on the image sample, wherein the computer-generated attention maps identify specific region containing a labeling error identified by the comparing;
presenting, by the one or more processors, the computer-generated attention maps on the display with the visual overlay and a personalized explanation of the labeling error to guide the human labeler in correcting the labeling error;
evaluating, by the one or more processors, a labeling performance of the human labeler based on the comparing using a weighting function and numeric metrics; and
selecting, by the one or more processors, a next image sample based on types of errors identified in the comparing, wherein the next image sample contains similar labeling challenges to reinforce training in areas where errors were identified.
2. The method of claim 1 , further comprising evaluating, by one or more processors, a labeling performance of the human labeler based on the comparing using a weighting function and numeric metrics.
3. The method of claim 2 , further comprising presenting, by one or more processors, image samples on the display for labeling by the human labeler once the labeling performance is above a threshold and contributing labeled image samples from the human labeler to a pool of image samples including image samples previously labeled by the expert human labeler or machine learning model.
4. The method of claim 3 , wherein the labeled image samples from the human labeler contributed to the pool include a weighting based on the labeling performance of the human labeler.
5. The method of claim 1 , further comprising certifying, by one or more processors, the human labeler for future labeling tasks when an labeling performance of the human labeler is above a predetermined level for labels of a type for which the human labeler has been trained.
6. The method of claim 1 , further comprising comparing, by one or more processors, labeling performances for multiple human labelers for a same group of images to establish a quality metric for the multiple human labelers.
7. The method of claim 1 , wherein presenting the computer-generated attention maps on the display to draw a human labeler attention to the labeling error identified by the comparing includes providing a personalized explanation of the labeling error on a display with the computer-generated attention maps.
8. The method of claim 1 , wherein the images to be labeled comprise at least one of medical images, geographic images, and industry images.
9. A human-model collaborative labeling system, comprising:
a database that stores images previously labeled by at least one of an expert human labeler and a machine learning model trained on expert-labeled images;
a display that displays images selected from the database;
a labeling system adapted to enable a human labeler to label images presented on the display; and
a labeling training system including one or more processors that:
selects an image sample from the database for display on the display for labeling by the human, labeler through a graphical user interface;
receives one or more proposed labels from the labeling system;
compares the one or more proposed labels to previous labels of the image sample by the expert human labeler or machine learning, model;
generates computer-generated attention maps comprising a visual overlay on the display to draw a human labeler attention to a specific region containing a labeling error identified by the comparing;
presents, by the one or more processors, the computer-generated attention maps on the display with the visual overlay and a personalized explanation of the labeling error to guide the human labeler in correcting the labeling error;
evaluates, by the one or more processors, a labeling performance of the human labeler based on the comparing using a weighting function and numeric metrics; and
selects a next image sample from the database based on any errors identified in the comparing;
wherein the computer-generated attention maps include a computer-generated visual overlay superimposed on the image sample, and wherein the next image sample contains similar labeling challenges to reinforce training in areas where errors were identified.
10. The system of claim 9 , wherein the labeling training system further evaluates an labeling performance of the human labeler by applying a weighting function and numeric metrics to comparison results from comparing the one or more proposed labels to previous labels of the image sample by the expert human labeler or machine learning model.
11. The system of claim 10 , wherein the labeling training system further presents image samples for labeling by the human labeler once the human labeler has been evaluated to have an labeling performance above a threshold and contributes labeled image samples from the human labeler to the database.
12. The system of claim 11 , wherein the labeled image samples from the human labeler contributed to the database include a weighting based on the labeling performance of the human labeler.
13. The system of claim 9 , wherein the labeling training system further certifies the human labeler for future labeling tasks when an labeling performance of the human labeler is above a predetermined level for labeling s of a type for which the human labeler has been trained.
14. The system of claim 9 , wherein the labeling training system further compares labeling performances for multiple human labelers for a same group of images to establish a quality metric for the multiple human labelers.
15. The system of claim 9 , wherein the labeling training system provides a personalized explanation of the labeling error on the display with the computer-generated attention maps.
16. The system of claim 9 , wherein the images to be labeled comprise at least one of medical images, geographic images, and industry images.
17. A non-transitory computer-readable medium storing computer instructions for training human labelers to label images, that when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, by one or more processors, a set of domain-specific images;
presenting, by the one or more processors, an image sample from the set of domain-specific images on a display to a human labeler for labeling via a graphical user interface, wherein the image sample has been previously labeled by at least one of an expert human and labeler or a machine learning model trained on expert-labeled images;
receiving, by the one or more processors, one or more proposed labels from the human labeler via the graphical user interface;
comparing the one or more proposed labels to previous labels of the image sample by the expert human labeler or machine learning model;
generating computer-generated attention maps comprising visual overlay superimposed on the image sample, wherein the computer-generated attention maps identify specific region containing a labeling error identified by the comparing;
presenting the computer-generated attention maps on the display with the visual overlay and a personalized explanation of the labeling error to guide the human labeler in correcting the labeling error;
evaluating a labeling performance of the human labeler based on the comparing using a weighting function and numeric metrics; and
selecting a next image sample from the set based on types of errors identified in the comparing, wherein the next image sample contains similar labeling challenges to reinforce training in areas where errors were identified.
18. The medium of claim 17 , further comprising instructions that when executed by the one or more processors cause the one or more processors to evaluate an labeling performance of the human labeler based on the comparing using a weighting function and numeric metrics.
19. The medium of claim 18 , further comprising instructions that when executed by the one or more processors cause the one or more processors to present image samples for labeling by the human labeler once the human labeler has been evaluated to have the labeling performance above a threshold and to contribute labeled image samples from the human labeler to a pool of image samples including image samples previously labeled by the expert human labeler or machine learning model.
20. The medium of claim 19 , wherein the labeled image samples from the human labeler contributed to the pool include a weighting based on the labeling performance of the human labeler.