Systems and methods for training image detection systems for augmented and mixed reality applications
Described are system, method, and computer-program product embodiments for developing an object detection model. The object detection model may detect a physical object in an image of a real world environment. A system can automatically generate a plurality of synthetic images. The synthetic images can be generated by randomly selecting parameters of the environmental features, camera intrinsics, and a target object. The system may automatically annotate the synthetic images to identify the target object. In some embodiments, the annotations can include information about the target object determined at the time the synthetic images are generated. The object detection model can be trained to detect the physical object using the annotated synthetic images. The trained object detection model can be validated and tested using at least one image of a real world environment. The image(s) of the real world environment may or may not include the physical object.
1. A method for training a model, the method comprising:
generating a plurality of synthetic images, the generation including selecting parameters of environmental features, camera intrinsics, and a target object, the target object being a simulation of a physical object, wherein the generation of the plurality of synthetic images includes generating a greater number of synthetic images assigned high probabilistic weights than low probabilistic weights, the assigned probabilistic weights representing a likelihood of the target object being associated with the selected parameters;
annotating the plurality of synthetic images with information related to properties of the target object; and
training the model to detect the physical object using the plurality of annotated synthetic images.
2. The method of claim 1 , wherein the generation of the plurality of synthetic images includes generating a number of synthetic images, wherein the number is not pre-defined.
3. The method of claim 1 , wherein each of the plurality of synthetic images has at least one parameter different from the others of the plurality of synthetic images.
4. The method of claim 3 , wherein the at least one parameter is one or more of: camera angle, lighting effects, and camera setting.
5. The method of claim 3 , wherein the at least one parameter is one or more of: target object size, target object color, and target object texture.
6. The method of claim 1 , wherein the annotation of the plurality of synthetic images includes using information from the generation of the plurality of synthetic images to define bounding boxes around the target object.
7. The method of claim 1 , wherein the annotation of the plurality of synthetic images includes adding class name and object location information to files representative of the plurality of synthetic images.
8. The method of claim 1 , wherein the training of the object detection model includes automatically identifying patterns in the plurality of annotated synthetic images that correlate to the targeted object.
9. The method of claim 1 , wherein the generation of the plurality of synthetic images includes not varying one or more target properties.
10. A method for using a model to detect a physical object, the method comprising:
receiving an image of a physical object;
using the model to detect the physical object, the object detection model being created by:
generating a plurality of synthetic images, the generation including selecting parameters of environmental features, camera intrinsics, and a target object, the target object being a simulation of the physical object, wherein the generation of the plurality of synthetic images includes generating a greater number of synthetic images assigned high probabilistic weights than low probabilistic weights, the assigned probabilistic weights representing a likelihood of the target object being associated with the selected parameters;
annotating the plurality of synthetic images with information related to properties of the target object; and
training the model to detect the physical object using the plurality of annotated synthetic images.
11. The method of claim 10 , wherein the model is used to instruct a user how to use, maintain, and/or repair a device.
12. A system for training a model, the system comprising one or more processors and a memory, wherein the one or more processors are configured to execute instructions stored on the memory to cause the system to:
generate a plurality of synthetic images, the generation including selecting parameters of environmental features, camera intrinsics, and a target object, the target object being a simulation of a physical object, wherein the generation of the plurality of synthetic images includes generating a greater number of synthetic images assigned high probabilistic weights than low probabilistic weights, the assigned probabilistic weights representing a likelihood of the target object being associated with the selected parameters;
annotate the plurality of synthetic images with information related to properties of the target object; and
train the model to detect the physical object using the plurality of annotated synthetic images.
13. The system of claim 12 , wherein the generation of the plurality of synthetic images includes generating a number of synthetic images, wherein the number is not pre-defined.
14. The system of claim 12 , wherein each of the plurality of synthetic images has at least one parameter different from the others of the plurality of synthetic images.
15. The system of claim 12 , wherein the training of the model includes automatically identifying patterns in the plurality of annotated synthetic images that correlate to the targeted object.
16. The system of claim 12 , wherein the generation of the plurality of synthetic images includes not varying one or more target properties.