Closed loop automatic dataset creation systems and methods
Various techniques are provided for training a neural network to classify images. A convolutional neural network (CNN) is trained using training dataset comprising a plurality of synthetic images. The CNN training process tracks image-related metrics and other informative metrics as the training dataset is processed. The trained inference CNN may then be tested using a validation dataset of real images to generate performance results (e.g., whether a training image was properly or improperly labeled by the trained inference CNN). In one or more embodiments, a training dataset and analysis engine extracts and analyzes the informative metrics and performance results, generates parameters for a modified training dataset to improve CNN performance, and generates corresponding instructions to a synthetic image generator to generate a new training dataset. The process repeats in an iterative fashion to build a final training dataset for use in training an inference CNN.
1. A system comprising:
a training dataset comprising a plurality of labeled images, wherein the plurality of labeled images comprises a plurality of real-world images and a plurality of synthetic images and wherein each labeled image includes a label identifying a correct object classification for a corresponding image;
a synthetic image generator operable to update the training dataset by generating a plurality of synthetic images of identified objects in virtual three-dimensional environments and corresponding labels identifying a correct classification for the identified objects in the corresponding synthetic images;
a neural network training system operable to train a convolutional neural network for object detection using the plurality of labeled images from the training dataset and produce a trained convolutional neural network; and
a training dataset analysis engine operable to analyze the trained neural network based at least in part on a validation process to generate parameters for an updated training dataset comprising an identification of a subset of the plurality of images to use in the updated training dataset based at least on part on objects correctly labeled during the validation process and parameters defining new synthetic images to be generated by the synthetic image generator based at least in part on objects with detected errors in classification;
wherein parameters defining new synthetic images to be generated comprise an identification of objects to be detected and classified, imaging sensors to be modeled, and/or three-dimensional virtual environments in which the new synthetic images are to be captured.
2. The system of claim 1 , wherein the neural network training system is further operable to generate informative metrics and/or performance results representing a contribution of the plurality of images to the neural network training system, the informative metrics and/or performance results comprising extracted features, classification labels, changes in neural network parameters, image characteristics and/or data from previous training dataset iterations; and
wherein the training dataset analysis engine is operable to generate parameters for new training data in response to the informative metrics, the parameters for new training data including labels, objects, and/or environments to be used for images for the updated training dataset.
3. The system of claim 1 , further comprising a validation system comprising a validation dataset comprising a plurality of labeled, real world images, wherein the trained neural network classifies images from the validation dataset and generates an error associated with an accuracy of image classification; and
wherein the training dataset analysis engine is operable to generate parameters for new training data in response to errors in the image classification.
4. The system of claim 1 , wherein the training dataset analysis engine comprises a data extractor/analyzer operable to receive informative metrics and performance results including extracted features, changes in neural network parameters and/or data from previous training dataset iterations and evaluate images based on a contribution to the neural network, wherein the contribution includes a ranking of the images based on the performance results.
5. The system of claim 1 , wherein the training dataset analysis engine comprises a dataset generator operable to define the updated training dataset comprising a subset of the plurality of labeled images and parameters for a new synthetic images to be generated; and
wherein the training dataset and analysis engine comprises an assembler/interface operable to generate instructions for the synthetic image generator to create the updated training dataset.
6. The system of claim 1 , further comprising configuration parameters defining a scope of an image classification system including labels, objects and/or environments to be used for labeled images in the training dataset.
7. The system of claim 1 , wherein the training dataset analysis engine is further operable to rank each of the plurality of images based on a relative contribution to a performance of the neural network; and wherein the subset of the plurality of images comprises images having a rank above a threshold.
8. The system of claim 1 , wherein the synthetic image generator is operable to receive the parameters defining new synthetic images, generate imaging scenarios including randomized object types and/or placement within a three-dimensional virtual environment; and
for each imaging scenario, generate a corresponding synthetic image for an updated training dataset.
9. The system of claim 1 , wherein the updated training dataset comprises a subset of the plurality of labeled images.
10. The system of claim 1 wherein the system is further operable to train the convolutional neural network using the plurality of images, validate the trained convolutional neural network using real images to detect classification errors, and update the parameters in response to detected classification errors to improve classification accuracy.
11. A method comprising:
providing a training dataset comprising a plurality of labeled images, wherein the plurality of labeled images comprises a plurality of real-world images and a plurality of synthetic images and wherein each labeled image includes a label identifying a correct object classification for a corresponding image;
generating, via a synthetic image generator, a plurality of synthetic images of identified objects in virtual three-dimensional environments and corresponding labels identifying a correct classification for the object in the corresponding synthetic images;
training, via a neural network training system, a convolutional neural network for object detection using the plurality of labeled images from the training dataset to produce a trained convolutional neural network; and
generating, based on an analysis of the trained neural network via a training dataset analysis engine, parameters for an updated training dataset comprising an identification of a subset of the plurality of labeled images and parameters defining new synthetic images to be generated by the synthetic image generator based at least in part on objects with detected errors in classification;
wherein parameters defining new synthetic images to be generated comprise an identification of objects to be detected and classified, imaging sensors to be modeled, and/or three-dimensional virtual environments in which the new synthetic images are to be captured.
12. The method of claim 11 , wherein training the convolutional neural network further comprises generating informative metrics representing a contribution of the plurality of images to the neural network training system, the informative metrics comprising extracted features, changes in neural network parameters, and/or data from previous training dataset iterations; and
wherein parameters for new training data is generated in response to the informative metrics, the parameters for the new training data comprising labels, objects, and/or environments to be used for images in the updated training dataset.
13. The method of claim 11 , further comprising validating, using a validation dataset comprising a plurality of labeled, real world images, the trained neural network to detect image classification errors; and
wherein parameters for new training data is generated in response to the image classification errors.
14. The method of claim 11 , further comprising receiving informative metrics and performance results including extracted features, changes in neural network parameters and/or data from previous training dataset iterations and evaluating images based on a contribution to the neural network, wherein the contribution includes a ranking of the images based on the performance results.
15. The method of claim 11 , further comprising defining the updated training dataset comprising a subset of the plurality of labeled images and parameters for a new synthetic images to be generated; and
generating instructions for the synthetic image generator to create the updated training dataset.
16. The method of claim 11 , further comprising determining configuration parameters defining a scope of a classification system, including labels, objects and/or environments to be used for labeled images in the training dataset.
17. The method of claim 11 , further comprising ranking each of the plurality of images based on a relative contribution to a performance of the neural network; and wherein the subset of the plurality of images comprises the images having a rank above a threshold.
18. The method claim 11 , further comprising receiving the parameters defining new synthetic images, generating imaging scenarios including randomized object types and/or placement within a three-dimensional virtual environment; and
generating, for each imaging scenario, a corresponding synthetic image for the updated training dataset.
19. The method of claim 11 , further comprising generating the updated training dataset comprising a subset of the plurality of images.
20. The method of claim 11 , further comprising training the convolutional neural network using the generated synthetic images, validating the training using real images to detect classification errors, and updating the parameters for the updated training dataset in response to detected classification errors to improve classification accuracy.