Systems and methods for pose detection and measurement
A method for estimating a pose of an object includes: receiving a plurality of images of the object captured from multiple viewpoints with respect to the object; initializing a current pose of the object based on computing an initial estimated pose of the object from at least one of the plurality of images; predicting a plurality of 2-D keypoints associated with the object from each of the plurality of images; and computing an updated pose that minimizes a cost function based on a plurality of differences between the 2-D keypoints and a plurality of 3-D keypoints associated with a 3-D model of the object as arranged in accordance with the current pose, and as projected to each of the viewpoints.
1. A method for estimating a pose of an object comprising:
receiving a plurality of images of the object captured from multiple viewpoints with respect to the object;
initializing a current pose of the object based on computing an initial estimated pose of the object from at least one of the plurality of images;
performing an instance segmentation process on the plurality of images captured from multiple viewpoints;
extracting an image patch for the object from an instance of the object detected by the image segmentation process;
predicting a plurality of 2-D keypoints associated with the object from each of the plurality of images including providing the extracted image patch to a keypoint detector; and
computing an updated pose that minimizes a cost function based on a plurality of differences between the 2-D keypoints and a plurality of 3-D keypoints corresponding to three-dimensional points on the surface of a 3-D model of the object as arranged in accordance with the current pose, and as the 3-D keypoints are projected to each of the viewpoints.
2. The method of claim 1 , wherein each of the plurality of differences corresponds to a different viewpoint of the viewpoints, and
wherein each of the differences is computed for a viewpoint of the viewpoints based on a difference between:
the plurality of 2-D keypoints associated with an image of the plurality of images corresponding to the viewpoint; and
projecting the 3-D keypoints of the 3-D model arranged in the current pose to the viewpoint.
3. The method of claim 2 , wherein the cost function is:
min
R
o
,
T
o
∑
i
:
N
,
j
:
M
c
i
j
[
u
ij
,
v
i
j
]
-
K
j
[
R
j
T
j
]
[
R
o
T
o
]
[
x
i
,
y
i
,
z
i
]
wherein R o , T o is the pose of the object, i is an index iterating through N 3-D keypoints, j is an index iterating through M viewpoints, c ij is a confidence for a corresponding keypoint prediction [u ij , V ij ] of an i-th keypoint in an image for corresponding to a j-th viewpoint, R j , T j , K j are extrinsic parameters and intrinsic parameters of a j-th camera used to capture an image from a j-th viewpoint, and [x i , y i , z i ] is an i-th 3-D keypoint.
4. The method of claim 2 , wherein the cost function further accounts for symmetries in the object in accordance with:
min
R
o
,
T
o
∑
i
:
N
,
j
:
M
min
S
~
v
j
c
i
j
[
u
ij
,
v
i
j
]
-
K
j
S
[
R
j
T
j
]
[
R
o
T
o
]
[
x
i
,
y
i
,
z
i
]
wherein R o , T o is the pose of the object, i is an index iterating through N 3-D keypoints, j is an index iterating through M viewpoints, c ij is a confidence for a corresponding keypoint prediction [u ij , v ij ] of an i-th keypoint in an image from a j-th viewpoint, R j , T j , K j are extrinsic parameters and intrinsic parameters of the j-th camera used to capture the images from a j-th viewpoint, [x i , y i , z i ] is an i-th 3-D keypoint, S is a transform between different symmetries of the object, and v is the j-th view.
5. The method of claim 1 , wherein the predicting the plurality of 2-D keypoints is performed using a deep learning keypoint detector.
6. The method of claim 5 , wherein the images comprise polarization information, and
wherein the deep learning keypoint detector is configured to compute predicted 2-D keypoints based on the polarization information of the images.
7. The method of claim 6 , wherein the deep learning keypoint detector is trained based on synthesized training data comprising synthesized polarization information and synthesized keypoint labels.
8. The method of claim 1 , wherein the images comprise polarization information, and
wherein the method further comprises refining the updated pose by aligning the 3-D model with the polarization information.
9. The method of claim 8 , wherein the refining the updated pose comprises:
detecting one or more edges of the object in the images to generate one or more measured edge images;
detecting correspondences between the one or more measured edges rendered edge images of the 3-D model arranged in the current pose; and
iteratively refining the updated pose using multi-view perspective-n-point based on the detected correspondences.
10. The method of claim 9 , wherein the edges are detected by:
computing surface normals based on the polarization information; and
detecting edges of the object at discontinuities in the surface normals.
11. The method of claim 9 , wherein the edges are detected by:
computing a plurality of instance segmentation maps from the images; and
detecting edges of the object at boundaries of instances of the object in the instance segmentation maps.
12. The method of claim 8 , wherein the refining the updated pose comprises:
computing measured surface normals of the object based on the polarization information;
placing the 3-D model of the object based on the updated pose; and
iteratively updating the pose of the 3-D model to align the surface normals of the 3-D model with the measured surface normals.
13. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving a plurality of images of the object captured from multiple viewpoints with respect to the object;
initializing a current pose of the object based on computing an initial estimated pose of the object from at least one of the plurality of images;
performing an instance segmentation process on the plurality of images captured from multiple viewpoints;
extracting an image patch for the object from an instance of the object detected by the image segmentation process;
predicting a plurality of 2-D keypoints associated with the object from each of the plurality of images including providing the extracted image patch to a keypoint detector; and
computing an updated pose that minimizes a cost function based on a plurality of differences between the 2-D keypoints and a plurality of 3-D keypoints corresponding to three-dimensional points on the surface of a 3-D model of the object as arranged in accordance with the current pose, and as the 3-D keypoints are projected to each of the viewpoints.
14. The system of claim 13 , wherein each of the plurality of differences corresponds to a different viewpoint of the viewpoints, and
wherein each of the differences is computed for a viewpoint of the viewpoints based on a difference between:
the plurality of 2-D keypoints associated with an image of the plurality of images corresponding to the viewpoint; and
projecting the 3-D keypoints of the 3-D model arranged in the current pose to the viewpoint.
15. The system of claim 14 , wherein the cost function is:
min
R
o
,
T
o
∑
i
:
N
,
j
:
M
c
ij
[
u
ij
,
v
ij
]
-
K
j
[
R
j
T
j
]
[
R
o
T
o
]
[
x
i
,
y
i
,
z
i
]
wherein R o , T o is the pose of the object, i is an index iterating through N 3-D keypoints, j is an index iterating through M viewpoints, c ij is a confidence for a corresponding keypoint prediction [u ij , v ij ] of an i-th keypoint in an image for corresponding to a j-th viewpoint, R j , T j , K j are extrinsic parameters and intrinsic parameters of a j-th camera used to capture an image from a j-th viewpoint, and [x i , y i , z i ] is an i-th 3-D keypoint.
16. The system of claim 14 , wherein the cost function further accounts for symmetries in the object in accordance with:
min
R
o
,
T
o
∑
i
:
N
,
j
:
M
min
S
∼
v
j
c
ij
[
u
ij
,
v
ij
]
-
K
j
S
[
R
j
T
j
]
[
R
o
T
o
]
[
x
i
,
y
i
,
z
i
]
wherein R o , T o is the pose of the object, i is an index iterating through N 3-D keypoints, j is an index iterating through M viewpoints, c ij is a confidence for a corresponding keypoint prediction [u ij , v ij ] of an i-th keypoint in an image from a j-th viewpoint, R j , T j , K j are extrinsic parameters and intrinsic parameters of the j-th camera used to capture the images from a j-th viewpoint, [x i , y i , z i ] is an i-th 3-D keypoint, S is a transform between different symmetries of the object, and v j is the j-th view.
17. One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:
receiving a plurality of images of the object captured from multiple viewpoints with respect to the object;
initializing a current pose of the object based on computing an initial estimated pose of the object from at least one of the plurality of images;
performing an instance segmentation process on the plurality of images captured from multiple viewpoints;
extracting an image patch for the object from an instance of the object detected by the image segmentation process;
predicting a plurality of 2-D keypoints associated with the object from each of the plurality of images including providing the extracted image patch to a keypoint detector; and
computing an updated pose that minimizes a cost function based on a plurality of differences between the 2-D keypoints and a plurality of 3-D keypoints corresponding to three-dimensional points on the surface of a 3-D model of the object as arranged in accordance with the current pose, and as the 3-D keypoints are projected to each of the viewpoints.
18. The non-transitory computer storage media of claim 17 , wherein each of the plurality of differences corresponds to a different viewpoint of the viewpoints, and
wherein each of the differences is computed for a viewpoint of the viewpoints based on a difference between:
the plurality of 2-D keypoints associated with an image of the plurality of images corresponding to the viewpoint; and
projecting the 3-D keypoints of the 3-D model arranged in the current pose to the viewpoint.
19. The non-transitory computer storage media of claim 18 , wherein the cost function is:
min
R
o
,
T
o
∑
i
:
N
,
j
:
M
c
ij
[
u
ij
,
v
ij
]
-
K
j
[
R
j
T
j
]
[
R
o
T
o
]
[
x
i
,
y
i
,
z
i
]
wherein R o , T o is the pose of the object, i is an index iterating through N 3-D keypoints, j is an index iterating through M viewpoints, c ij is a confidence for a corresponding keypoint prediction [u ij , v ij ] of an i-th keypoint in an image for corresponding to a j-th viewpoint, R j , T j , k j are extrinsic parameters and intrinsic parameters of a j-th camera used to capture an image from a j-th viewpoint, and [x i , y i , z i ] is an i-th 3-D keypoint.
20. The non-transitory computer storage media of claim 18 , wherein the cost function further accounts for symmetries in the object in accordance with:
min
R
o
,
T
o
∑
i
:
N
,
j
:
M
min
S
∼
v
j
c
ij
[
u
ij
,
v
ij
]
-
K
j
S
[
R
j
T
j
]
[
R
o
T
o
]
[
x
i
,
y
i
,
z
i
]
wherein R o , T o is the pose of the object, i is an index iterating through N 3-D keypoints, j is an index iterating through M viewpoints, c ij is a confidence for a corresponding keypoint prediction [u ij , v ij ] of an i-th keypoint in an image from a j-th viewpoint, R j , T j , K j are extrinsic parameters and intrinsic parameters of the j-th camera used to capture the images from a j-th viewpoint, [x i , y i , z i ] is an i-th 3-D keypoint, S is a transform between different symmetries of the object, and v j is the j-th view.