Vision data enrichment and annotation platform
A platform for data collection and annotation is disclosed. The systems and methods herein describe a system that can receive vision data from a plurality of heterogenous sources and store this vision data in a common datastore. The vision data is then processed (for example, prioritized and annotated) at common applications, while security and access rights features are maintained. Machine learning models for annotation are trained in such a way as to generate an efficient and accurate model, while ensuring that manual annotation costs remain below a set budget.
1 . A computing system, comprising:
at least one processor; and
at least one memory storing computer-executable instructions, the computer-executable instructions when executed by the at least one processor causing the computer to:
receive:
a plurality of images to be annotated at a machine learning platform trained using a training data set, the training data set comprising annotated image data, and
a budget for the annotation of the plurality of images;
automatically annotate, at the machine learning platform, each of the plurality of images, wherein each annotated image of the plurality of images has an associated annotation confidence score;
select a first set of images of the plurality of images, the first set of images being lesser in quantity than the plurality of images, the first set of images having an annotation confidence score below a threshold;
receive user-directed re-annotated images of each of the first set of images, the re-annotated images having an annotation confidence score above the threshold, wherein annotating each image of the first set of images by a user draws from the budget;
train the machine learning platform using the re-annotated images;
randomly select a second set of images of the plurality of images, the second set of images being lesser in quantity than the plurality of images;
validate a resulting machine learning model using a first validation set, wherein validating the resulting machine learning model comprises:
receiving user-directed annotated images of each of the second set of images, the annotated second set of images being designated as the first validation set, wherein annotating each image of the second set of images by the user draws from the budget;
automatically annotating, at the machine learning platform, each of the second set of images; and
comparing the automatically annotated second set of images and the first validation set;
update the machine learning platform based on the validation;
determine a remaining amount of the budget;
based on the remaining amount of the budget:
randomly select a test set of images of the plurality of images, the test set of images being lesser in quantity than the plurality of images;
receive user-directed annotated images of each of the test set of images, wherein annotating each image of the test set of images by the user draws from the budget;
automatically annotate, at the updated machine learning platform, each annotated image of the annotated test set of images, thereby testing the machine learning model;
calculate final annotation confidence scores for each of the annotated test set of images; and
automatically annotate, at the machine learning platform, utilizing the tested machine learning model, each image of the plurality of images that was not included in a previously selected set of images for annotation.
2 . The computing system of claim 1 , further comprising instructions to:
wherein the remaining amount of budget indicates that the budget is sufficient to further train and validate the machine learning model, and prior to selecting the test set of images, select a third set of images of the plurality of images, the third set of images being lesser in quantity than the plurality of images, the third set of images having a confidence score below the threshold;
receive user-directed annotated images of each of the third set of images, the re-annotated images having a confidence score above the threshold, wherein annotating each image of the third set of images by the user draws from the budget;
train the machine learning platform using the re-annotated images;
randomly select a fourth set of images of the plurality of images, the fourth set of images being lesser in quantity than the plurality of images;
validate the resulting machine learning model using a second validation set, wherein validating the resulting machine learning model comprises;
receiving user-directed annotated images of each of the fourth set of images, the annotated fourth set of images being designated as the second validation set, wherein annotating each image of the fourth set of images by the user draws from the budget;
automatically annotating, at the machine learning platform, each of the fourth set of images; and
comparing the automatically annotated fourth set of images and the second validation set;
update the machine learning platform based on the validation; and
determine that the budget includes only a quantity of budget sufficient to annotate the test set of images.
3 . The computing system of claim 1 , wherein the user-directed annotated images are received from an annotation tool operated by an annotation user via an annotation user interface.
4 . The computing system of claim 1 , wherein the plurality of images represents a set of images of a plurality of sets of images, wherein each set of images has an assigned annotation priority.
5 . The computing system of claim 1 , wherein the machine learning platform is configured to generate an object recognition machine learning model.
6 . The computing system of claim 1 , wherein the machine learning platform is configured to generate a text recognition machine learning model.
7 . The computing system of claim 1 , wherein each of the plurality of images has an associated context, and wherein automatic annotation is based at least in part on the context.
8 . A method, comprising:
receiving a plurality of images to be annotated at a machine learning platform trained using a training data set, the training data set comprising annotated image data;
automatically annotating, at the machine learning platform, each of the plurality of images, wherein each annotated image of the plurality of images has an associated annotation confidence score;
selecting a first set of images of the plurality of images, the first set of images being lesser in quantity than the plurality of images, the first set of images having an annotation confidence score below a threshold;
receiving user-directed re-annotated images of each of the first set of images, the re-annotated images having an annotation confidence score above the threshold, wherein annotating each image of the first set of images by the user draws from a budget;
training the machine learning platform using the re-annotated images;
randomly selecting a second set of images of the plurality of images, the second set of images being lesser in quantity than the plurality of images;
validating a resulting machine learning model using a first validation set, wherein validating the resulting machine learning model comprises:
receiving user-directed annotated images of each of the second set of images, the annotated second set of images being designated as the first validation set, wherein annotating each image of the second set of images by the user draws from the budget;
automatically annotating, at the machine learning platform, each of the second set of images; and
comparing the automatically annotated second set of images and the first validation set;
updating the machine learning platform based on the validation;
determining a remaining amount of the budget;
based on the remaining amount of the budget:
randomly selecting a test set of images of the plurality of images, the test set of images being lesser in quantity than the plurality of images;
receiving user-directed annotated images of each of the test set of images, wherein annotating each image of the test set of images by the user draws from the budget;
automatically annotating, at the updated machine learning platform, each annotated image of the annotated test set of images, thereby testing the machine learning model;
calculating final annotation confidence scores for each of the annotated test set of images; and
automatically annotating, at the machine learning platform, utilizing the tested machine learning model, each image of the plurality of images that was not included in a previously selected set of images for annotation.
9 . The method of claim 8 , further comprising:
wherein the remaining amount of budget indicates that the budget is sufficient to further train and validate the machine learning model, and prior to selecting the test set of images, selecting a third set of images of the plurality of images, the third set of images being lesser in quantity than the plurality of images, the third set of images having a confidence score below the threshold;
receiving user-directed annotated images of each of the third set of images, the re-annotated images having a confidence score above the threshold, wherein annotating each image of the third set of images by the user draws from the budget;
training the machine learning platform using the re-annotated images;
randomly selecting a fourth set of images of the plurality of images, the fourth set of images being lesser in quantity than the plurality of images;
validating the resulting machine learning model using a second validation set, wherein validating the resulting machine learning model comprises;
receiving user-directed annotated images of each of the fourth set of images, the annotated fourth set of images being designated as the second validation set, wherein annotating each image of the fourth set of images by the user draws from the budget;
automatically annotating, at the machine learning platform, each of the fourth set of images; and
comparing the automatically annotated fourth set of images and the second validation set;
updating the machine learning platform based on the validation; and
determining that the budget includes only a quantity of budget sufficient to annotate the test set of images.
10 . The method of claim 8 , wherein the user-directed annotated images are received from an annotation tool operated by an annotation user via an annotation user interface.
11 . The method of claim 8 , wherein the plurality of images represents a set of images of a plurality of sets of images, wherein each set of images has an assigned annotation priority.
12 . The method of claim 8 , wherein the machine learning platform is configured to generate an object recognition machine learning model.
13 . The method of claim 8 , wherein the machine learning platform is configured to generate a text recognition machine learning model.
14 . The method of claim 8 , wherein each of the plurality of images has an associated context, and wherein automatic annotation is based at least in part on the context.
15 . A computing system, comprising:
at least one processor; and
at least one memory storing computer-executable instructions, the computer-executable instructions when executed by the at least one processor causing the computing system to:
receive an annotation request at an annotation tool, the annotation request identifying a model and a training data set including a plurality of images to be annotated;
automatically annotate, at a machine learning platform, each of the plurality of images in the training data set, wherein each annotated image of the plurality of images has an associated annotation confidence score;
select a first subset of images of the plurality of images having annotation confidence scores below a threshold;
receive user-directed re-annotation of each of the first set of images, wherein the user-directed re-annotation of each image of the first set of images by a user draws from a budget;
retrain the model at the machine learning platform using the plurality of images in the training data set including a portion of the automatically annotated images having annotation confidence scores above the threshold and the re-annotated images to form a retrained model;
select a second subset of images of the plurality of images;
receive user-directed re-annotation of each of the second set of images;
validate the retrained model by comparing the user-directed re-annotation of each of the second set of images against automatic annotations of each of the second set of images generated using the retrained model;
determine a remaining amount of the budget;
based on the remaining amount of the budget, determine whether to iteratively select a further subset of the plurality of images for user-directed re-annotation; and
automatically annotate, at the machine learning platform, utilizing the retrained model, each image of a plurality of images that was not included in a previously selected set of images for annotation.
16 . The computing system of claim 15 , wherein the computing system is further configured to: based on the remaining amount of the budget falling below a threshold, calculate final annotation confidence scores for each of the plurality of images.
17 . The computing system of claim 15 , wherein the model is a context specific model and the plurality of images are associated with a context of the context-specific model.