Systems and methods for using computer vision to pick up small objects
A method for estimating a pose of an object includes: receiving, by a processor, an observed image depicting the object from a viewpoint; computing, by the processor, an instance segmentation map identifying a class of the object depicted in the observed image; loading, by the processor, a 3-D model corresponding to the class of the object; computing, by the processor, a rendered image of the 3-D model in accordance with an initial pose estimate of the object and the viewpoint of the observed image; computing, by the processor, a plurality of dense image-to-object correspondences between the observed image of the object and the 3-D model based on the observed image and the rendered image; and computing, by the processor, the pose of the object based on the dense image-to-object correspondences.
1. A method comprising:
receiving, by a processor, an observed image depicting a plurality of objects from a viewpoint;
computing, by the processor, an instance segmentation map identifying a class of the plurality of objects depicted in the observed image;
loading, by the processor, a 3-D model corresponding to the identified class of the plurality of objects, wherein the plurality of objects are homogeneous objects of the same identified class;
computing, by the processor, a rendered image comprising a plurality of renderings of the plurality objects based on the 3-D model in accordance with respective corresponding initial pose estimates of the plurality of objects and the viewpoint of the observed image;
computing, by the processor, a plurality of dense image-to-object correspondences between the observed image of the plurality of objects and the 3-D model based on the observed image and the rendered image; and
computing, by the processor, a plurality of poses of the plurality of objects based on the dense image-to-object correspondences.
2. The method of claim 1 , wherein the rendered image of the 3-D model is a rendered surface normals map.
3. The method of claim 2 , wherein the observed image is a color image.
4. The method of claim 2 , wherein the observed image is an observed surface normals map.
5. The method of claim 4 , wherein the observed surface normals map is computed from polarization raw frames captured of the object.
6. The method of claim 1 , wherein computing the dense image-to-object correspondences comprises:
computing a plurality of optical flow features from a plurality of pixels of the observed image to pixels of the rendered image; and
mapping the pixels of the rendered image to the renderings based on the 3-D model.
7. The method of claim 1 , wherein the computing the dense image-to-object correspondences comprises:
computing a plurality of dense disparity matches from a plurality of pixels of the observed image to pixels of the rendered image; and
mapping the pixels of the rendered image to the renderings based on the 3-D model.
8. The method of claim 1 , wherein the plurality of poses of the objects are computed by supplying the dense image-to-object correspondences as input to a Perspective-n-Point (PnP) algorithm.
9. The method of claim 8 , wherein the PnP algorithm is configured to compute each pose by identifying a pose P that minimizes an error function:
arg
min
P
∑
x
∈
X
KP
f
(
x
)
-
x
wherein K is a camera intrinsic matrix of a camera used to capture the observed image, P is a pose matrix representing a transformation between the pose of the object and the camera, f: 2 → 3 represents the dense image-to-object correspondences from pixels of the observed image to coordinates of the 3-D model, and X is the domain of f comprising the pixels of the observed image.
10. The method of claim 8 , wherein computing each pose of the plurality of objects further comprises computing a deformation of the 3-D model to match a configuration of the object as:
{ Pf ( x )−proj L(x) ( Pf ( x ))| x∈X}
where L(x) represents a line of projection of point x from a camera used to capture the observed image, P is a pose matrix representing a transformation between the pose of the object and the camera, f: 2 → 3 represents the dense image-to-object correspondences from pixels of the observed image to coordinates of the 3-D model, proj L(x) (Pf(x)) is an estimated depth of the object at point x from the camera along line L(x), and X is the domain of f comprising the pixels of the observed image.
11. The method of claim 8 , wherein the observed image comprises a depth map D, and
wherein the computing each pose of the plurality of objects further comprises computing a deformation of the 3-D model to match a configuration of the object as:
{ Pf ( x )− D ( x )| x∈X}
wherein D(x) is a depth of a pixel x as represented in the depth map D, P is a pose matrix representing a transformation between the pose of the object and a camera used to capture the observed image, f: 2 → 3 represents the dense image-to-object correspondences from pixels of the observed image to coordinates of the 3-D model, and X is the domain of f comprising the pixels of the observed image.
12. The method of claim 1 , wherein the method is a portion of a controller pipeline configured to control a robotic system.
13. A system comprising:
a camera; and
one or more computers coupled to the camera and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving an observed image from the camera, the observed image depicting a plurality of objects from a viewpoint of the camera;
computing an instance segmentation map identifying a class of the plurality of objects depicted in the observed image;
loading a 3-D model corresponding to the identified class of the plurality of objects, wherein the plurality of objects are homogeneous objects of the same identified class;
computing a rendered image comprising a plurality of renderings of the plurality of objects based on the 3-D model in accordance with respective corresponding initial pose estimates of the plurality of objects and the viewpoint of the observed image;
computing a plurality of dense image-to-object correspondences between the observed image of the plurality of objects and the 3-D model based on the observed image and the rendered image; and
computing a plurality of poses of the plurality of objects based on the dense image-to-object correspondences.
14. The system of claim 13 , wherein the rendered image of the 3-D model is a rendered surface normals map.
15. The system of claim 14 , wherein the observed image is a color image.
16. The system of claim 14 , wherein the observed image is an observed surface normals map.
17. The system of claim 16 , wherein the observed surface normals map is computed from polarization raw frames captured of the object.
18. The system of claim 13 , wherein the operations further comprise computing the dense image-to-object correspondences including:
computing a plurality of optical flow features from a plurality of pixels of the observed image to pixels of the rendered image; and
mapping the pixels of the rendered image to the renderings based on the 3-D model.
19. The system of claim 13 , wherein the operations further comprise computing the dense image-to-object correspondences including:
computing a plurality of dense disparity matches from a plurality of pixels of the observed image to pixels of the rendered image; and
mapping the pixels of the rendered image to the renderings based on the 3-D model.
20. The system of claim 13 , wherein the operations further comprise computing the plurality of poses of the objects by supplying the dense image-to-object correspondences as input to a Perspective-n-Point (PnP) algorithm.
21. The system of claim 20 , wherein the PnP algorithm is configured to compute each pose by identifying a pose P that minimizes an error function:
arg
min
P
∑
x
∈
X
KP
f
(
x
)
-
x
wherein K is a camera intrinsic matrix of the camera, P is a pose matrix representing a transformation between the pose of the object and the camera, f: 2 → 3 represents the dense image-to-object correspondences from pixels of the observed image to coordinates of the 3-D model, and X is the domain of f comprising the pixels of the observed image.
22. The system of claim 20 , wherein computing each pose of the plurality of objects comprises computing a deformation of the 3-D model to match a configuration of the object as:
{ Pf ( x )−proj L(x) ( Pf ( x ))| x∈x}
where L(x) represents a line of projection of point x from the camera, P is a pose matrix representing a transformation between the pose of the object and the camera, f: 2 → 3 represents the dense image-to-object correspondences from pixels of the observed image to coordinates of the 3-D model, proj L(x) (Pf(x)) is an estimated depth of the object at point x from the camera along line L(x), and X is the domain of f comprising the pixels of the observed image.
23. The system of claim 20 , wherein the observed image comprises a depth map D, and
wherein computing each pose of the plurality of objects comprises computing a deformation of the 3-D model to match a configuration of the object as:
{ Pf ( x )− D ( x )| x∈X}
wherein D(x) is a depth of a pixel x as represented in the depth map D, P is a pose matrix representing a transformation between the pose of the object and the camera, f: 2 → 3 represents the dense image-to-object correspondences from pixels of the observed image to coordinates of the 3-D model, and X is the domain of f comprising the pixels of the observed image.
24. The system of claim 13 , wherein the processor and memory are components of a controller configured to control a robotic system.
25. A method comprising:
receiving, by a processor, an observed image depicting a plurality of objects from a viewpoint;
computing, by the processor, an instance segmentation map identifying a respective class of each object of the plurality of objects depicted in the observed image;
loading, by the processor, a plurality of 3-D models corresponding respectively to the identified classes of the plurality of objects, wherein the objects are heterogeneous objects of a plurality of different classes;
computing, by the processor, a rendered image comprising a plurality of renderings of the plurality objects based on the plurality of 3-D models in accordance with respective corresponding initial pose estimates of the plurality of objects and the viewpoint of the observed image;
computing, by the processor, a plurality of dense image-to-object correspondences between the observed image of the plurality of objects and the plurality of 3-D models based on the observed image and the rendered image; and
computing, by the processor, a plurality of poses of the plurality of objects based on the dense image-to-object correspondences.
26. The method of claim 25 , wherein the rendered image of the 3-D model is a rendered surface normals map.
27. The method of claim 26 , wherein the observed image is an observed surface normals map.
28. The method of claim 27 , wherein the observed surface normals map is computed from polarization raw frames.
29. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving an observed image depicting a plurality of objects from a viewpoint;
computing an instance segmentation map identifying a respective class of each object of the plurality of objects depicted in the observed image;
loading a plurality of 3-D models corresponding respectively to the identified classes of the plurality of objects, wherein the objects are heterogeneous objects of a plurality of different classes;
computing a rendered image comprising a plurality of renderings of the plurality objects based on the plurality of 3-D models in accordance with respective corresponding initial pose estimates of the plurality of objects and the viewpoint of the observed image;
computing a plurality of dense image-to-object correspondences between the observed image of the plurality of objects and the plurality of 3-D models based on the observed image and the rendered image; and
computing a plurality of poses of the plurality of objects based on the dense image-to-object correspondences.
30. The system of claim 29 , wherein the rendered image of the 3-D model is a rendered surface normals map.
31. The system of claim 30 , wherein the observed image is an observed surface normals map.
32. The system of claim 31 , wherein the observed surface normals map is computed from polarization raw frames.