Systems and methods for generating and using visual datasets for training computer vision models
A system for collecting data for training a computer vision model for shape estimation includes: an imaging system configured to capture one or more images; and a processing system including a processor and memory storing instructions that, when executed by the processor, cause the processor to: receive one or more input images from the imaging system; estimate a pose of an object depicted in the one or more images; render a shape estimate from a 3-D model of the object posed in accordance with the pose of the object; and generate a data point of a training dataset, the data point including one or more images based on the one or more input images and a label corresponding to the one or more images, the label including the shape estimate.
1. A system for collecting data for a training dataset for training a machine learning model, the system comprising:
an imaging system configured to capture one or more images; and
a processing system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
for each of multiple iterations:
receiving one or more respective input images of a respective scene from the imaging system;
estimating a respective pose of a respective object in the respective scene from the one or more respective input images including aligning keypoints in the one or more respective input images with respective corresponding keypoints of a respective 3-D model of the respective object;
rendering, using the respective 3-D model of the respective object posed in accordance with the respective estimated pose, a respective shape estimate comprising a surface normals map that associates each of a plurality of locations in a respective input image with surface normal data;
generating a respective data point, the respective data point comprising a respective image of the respective scene and a respective label comprising the respective shape estimate of the respective object; and
including the respective data point in the training dataset;
training the machine learning model on the training dataset generated using the respective 3-D models of the respective objects to compute shape estimates based on one or more input images;
obtaining one or more images of a scene;
generating a shape estimate of an object in the scene from the one or more images using the machine learning model;
updating an estimated pose of the object to reduce a difference between the shape estimate and a rendered shape of a 3-D model of the object; and
providing the estimated pose to a controller for causing a robot to manipulate the object.
2. The system of claim 1 , wherein the imaging system comprises a polarization camera system, and
wherein the one or more respective input images comprise one or more polarization images,
wherein, for one or more of the multiple iterations, the operations further comprise generating a surface normals map from the one or more polarization images, and
wherein aligning the keypoints in the one or more respective input images with respective corresponding keypoints of the respective 3-D model of the respective object comprises refining the respective estimated pose of the object according to differences between the surface normals map computed from the one or more polarization images and surface normals of the respective 3-D model of the object.
3. The system of claim 2 wherein the one or more polarization images comprise a plurality of spectral channels corresponding to different portions of an electromagnetic spectrum.
4. The system of claim 1 , wherein the respective data point comprises one or more polarization images.
5. The system of claim 1 , wherein the respective data point comprises one or more polarization signatures computed based on one or more polarization images.
6. The system of claim 1 , wherein the respective data point comprises one or more surface normals maps computed from one or more polarization images.
7. The system of claim 2 , wherein the respective shape estimate comprises a rendered depth map.
8. The system of claim 1 , wherein the imaging system comprises a depth camera system, and
wherein the respective data point comprises one or more depth maps.
9. The system of claim 8 , wherein the respective pose of the respective object is estimated based on aligning a shape of the respective 3-D model with the one or more depth maps.
10. The system of claim 1 , wherein for one or more of the multiple iterations, estimating the respective pose of the respective object further comprises using a previously trained machine learning model trained to compute shape estimates based on the one or more respective input images, and wherein training the machine learning model on the training dataset comprises re-training the machine learning model on the respective generated data points.
11. The system of claim 1 , wherein training the machine learning model on the training dataset comprises training the machine learning model on the respective generated data points.
12. A method for collecting data for a training dataset for training a machine learning model, the method comprising:
for each of multiple iterations:
receiving, by a processing system comprising one or more processors and memory, one or more respective input images of a respective scene;
estimating, by the processing system, a respective pose of a respective object in the respective scene from the one or more respective input images including aligning keypoints in the one or more respective input images with respective corresponding keypoints of a respective 3-D model of the respective object;
rendering, by the processing system using the respective 3-D model of the respective object posed in accordance with the respective estimated pose, a respective shape estimate comprising a surface normals map that associates each of a plurality of locations in a respective input image with surface normal data;
generating, by the processing system, a respective data point, the respective data point comprising a respective image of the respective scene and a respective label comprising the respective shape estimate of the respective object; and
including the respective data point in the training dataset;
training the machine learning model on the training dataset generated using the respective 3-D models of the respective objects to compute shape estimates based on one or more input images;
obtaining one or more images of a scene;
generating a shape estimate of an object in the scene from the one or more images using the machine learning model;
updating an estimated pose of the object to reduce a difference between the shape estimate and a rendered shape of a 3-D model of the object; and
providing the estimated pose to a controller for causing a robot to manipulate the object.
13. The method of claim 12 , wherein the one or more respective input images comprise one or more polarization images, and wherein, for one or more of the multiple iterations, the method further comprises generating a surface normals map from the one or more polarization images,
wherein aligning the keypoints in the one or more respective input images with respective corresponding keypoints of the respective 3-D model of the respective object comprises refining the respective estimated pose of the respective object according to differences between the surface normals map computed from the one or more polarization images and surface normals of the respective 3-D model of the respective object.
14. The method of claim 13 , wherein the one or more polarization images comprise a plurality of spectral channels corresponding to different portions of an electromagnetic spectrum.
15. The method of claim 12 , wherein the respective data point comprises one or more polarization images.
16. The method of claim 12 , wherein the respective data point comprises one or more polarization signatures computed based on one or more polarization images.
17. The method of claim 12 , wherein the respective data point comprises one or more surface normals maps computed from one or more polarization images.
18. The method of claim 13 , wherein the respective shape estimate comprises a rendered depth map.
19. The method of claim 12 , wherein the respective data point comprises one or more depth maps.
20. The method of claim 19 , wherein the respective pose of the respective object is estimated based on aligning a shape of the respective 3-D model with the one or more depth maps.
21. The method of claim 12 , wherein, for one or more of the multiple iterations, estimating the respective pose of the respective object further comprises using a previously trained machine learning model trained to compute shape estimates based on the one or more respective input images, and wherein training the machine learning model on the training dataset comprises re-training the machine learning model on the respective generated data points.
22. The method of claim 12 , wherein training the machine learning model on the training dataset comprises training the machine learning model on the respective data points.
23. One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations for collecting data for a training dataset, comprising:
for each of multiple iterations:
receiving one or more respective input images of a respective scene;
estimating a respective pose of a respective object in the respective scene from the one or more respective input images including aligning keypoints in the one or more respective input images with respective corresponding keypoints of a respective 3-D model of the respective object;
rendering, using the respective 3-D model of the respective object posed in accordance with the respective estimated pose, a respective shape estimate comprising a surface normals map that associates each of a plurality of locations in a respective input image with surface normal data;
generating a respective data point, the respective data point comprising a respective image of the respective scene and a respective label comprising the respective shape estimate of the respective object; and
including the respective data point in the training dataset;
training a machine learning model on the training dataset generated using the respective 3-D models of the respective objects to compute shape estimates based on one or more input images;
obtaining one or more images of a scene;
generating a shape estimate of an object in the scene from the one or more images using the machine learning model;
updating an estimated pose of the object to reduce a difference between the shape estimate and a rendered shape of a 3-D model of the object; and
providing the estimated pose to a controller for causing a robot to manipulate the object.