Data synthesis using three-dimensional modeling
Techniques for data synthesis for training datasets for machine learning applications are described. A first image of at least an object from a first viewpoint is obtained. The first image having associated first image metadata including a first location of a feature of the object in the first image. A model is generated from the first image, the model including a three-dimensional representation of the object. A second image is generated from the model, the second image including the object from a second viewpoint that is different from the first viewpoint. Second image metadata is generated, the second image metadata including a second location of the feature of the object in the second image, the second location corresponding to the first location adjusted for the difference between the second viewpoint and the first viewpoint.
1. A computer-implemented method comprising:
obtaining a first image of at least an object from a first viewpoint, the first image having associated first image metadata including a first location of a feature of the object in the first image;
generating, from the first image, a model including a three-dimensional representation of the object;
generating a second image from the model, the second image including the object from a second viewpoint that is different from the first viewpoint; and
generating second image metadata, the second image metadata including a second location of the feature of the object in the second image, wherein the second location corresponds to the first location adjusted for the difference between the second viewpoint and the first viewpoint.
2. The computer-implemented method of claim 1 , wherein an illumination model lights the model for generating the second image, wherein the illumination model includes at least one of a light source position and a light source intensity.
3. The computer-implemented method of claim 2 , further comprising estimating an initial orientation of the object in the first image using a bilateral symmetry plane of the three-dimensional representation.
4. The computer-implemented method of claim 3 , wherein the bilateral symmetry plane is constrained to rotate away from the first viewpoint in the second viewpoint.
5. The computer-implemented method of claim 1 , wherein the model includes a background plane and further comprising applying pixel data from a portion of the first image not including the object to the background plane of the model.
6. The computer-implemented method of claim 5 , further comprising applying pixel data from a region including the object in the first image to the three-dimensional representation of the object to texture the three-dimensional representation of the object.
7. The computer-implemented method of claim 5 , wherein the second image includes at least a portion of the background plane.
8. The computer-implemented method of claim 1 , wherein the second viewpoint is based on a user-specified parameter received via an application programming interface.
9. The computer-implemented method of claim 1 , wherein the first image is the only image used to generate the three-dimensional representation.
10. A computer-implemented method comprising:
obtaining a first image and first image metadata, wherein the first image is of at least a portion of a human face from a first viewpoint, the first image metadata including a first location of a feature of the human face in the first image;
generating, from the first image, a model including a three-dimensional representation of the human face;
applying pixel data from the first image to the three-dimensional representation of the human face to generate a textured three-dimensional representation of the human face;
generating a second image using the textured three-dimensional representation of the human face, the second image including the human face from a second viewpoint that is different than the first viewpoint;
generating second image metadata, the second image metadata including a second location of the feature of the human face in the second image, wherein the second location corresponds to the first location adjusted for the difference between the second viewpoint and the first viewpoint; and
storing the second image and second image metadata in an image dataset.
11. The computer-implemented method of claim 10 , wherein the model includes a background plane that includes a background of the first image and wherein the second image includes a background based at least in part on the background plane.
12. The computer-implemented method of claim 10 , further comprising estimating an initial orientation of the human face in the first image using a bilateral symmetry plane of the three-dimensional representation, and wherein the bilateral symmetry plane is rotated away from the first viewpoint in the second viewpoint.
13. A system comprising:
a data store implemented by a first one or more electronic devices; and
a data synthesis service implemented by a second one or more electronic devices, the data synthesis service including instructions that upon execution cause the data synthesis service to:
obtain a first image from the data store, the first image of at least an object from a first viewpoint, the first image having associated first image metadata including a first location of a feature of the object in the first image;
generate, from the first image, a model including a three-dimensional representation of the object;
generate a second image from the model, the second image including the object from a second viewpoint that is different from the first viewpoint;
generate second image metadata, the second image metadata including a second location of the feature of the object in the second image, wherein the second location corresponds to the first location adjusted for the difference between the second viewpoint and the first viewpoint; and
store the second image and the second image metadata in the data store.
14. The system of claim 13 , wherein an illumination model lights the model for generating the second image, wherein the illumination model includes at least one of a light source position and a light source intensity.
15. The system of claim 13 , further comprising estimating an initial orientation of the object in the first image using a bilateral symmetry plane of the three-dimensional representation.
16. The system of claim 15 , wherein the bilateral symmetry plane is constrained to rotate away from the first viewpoint in the second viewpoint.
17. The system of claim 13 , wherein the model includes a background plane and further comprising applying pixel data from a portion of the first image not including the object to the background plane of the model.
18. The system of claim 17 , wherein the data synthesis service includes further instructions that upon execution cause the data synthesis service to apply pixel data from a region including the object in the first image to the three-dimensional representation of the object to texture the three-dimensional representation of the object.
19. The system of claim 17 , wherein the second image includes at least a portion of the background plane.
20. The system of claim 13 , further comprising:
a machine learning service implemented by a third one or more electronic devices, the machine learning service including instructions that upon execution cause the machine learning service to train a machine learning model using an image dataset, the image dataset including the first image, the second image, the first image metadata, and the second image metadata.