Three-dimensional (3D) pose estimation from a monocular camera
Estimating a three-dimensional (3D) pose of an object, such as a hand or body (human, animal, robot, etc.), from a 2D image is necessary for human-computer interaction. A hand pose can be represented by a set of points in 3D space, called keypoints. Two coordinates (x,y) represent spatial displacement and a third coordinate represents a depth of every point with respect to the camera. A monocular camera is used to capture an image of the 3D pose, but does not capture depth information. A neural network architecture is configured to generate a depth value for each keypoint in the captured image, even when portions of the pose are occluded, or the orientation of the object is ambiguous. Generation of the depth values enables estimation of the 3D pose of the object.
1. A computer-implemented method, comprising:
receiving locations of keypoints for a three-dimensional (3D) object, wherein each location includes pixel coordinates and a normalized depth value, the pixel coordinates corresponding to pixels within a two-dimensional (2D) image of the 3D object, the 2D image associated with camera attributes and the normalized depth values corresponding to normalized relative depth values of each one of the keypoints with respect to a reference keypoint;
computing a depth of the reference keypoint with respect to a camera based on the locations and the camera attributes; and
computing a scale normalized 3D pose of the 3D object based on the locations, the depth of the reference keypoint and the camera attributes.
2. The computer-implemented method of claim 1 , further comprising receiving a scale factor corresponding to the 3D object.
3. The computer-implemented method of claim 2 , further comprising generating, according to the scale factor, an absolute 3D pose of the 3D object from the scale normalized 3D pose.
4. The computer-implemented method of claim 2 , wherein the scale factor is estimated and corresponds to a component of the 3D object.
5. The computer-implemented method of claim 2 , wherein the scale factor is measured and corresponds to a component of the 3D object.
6. The computer-implemented method of claim 2 , wherein the scale factor is a distance between two vertices or two keypoints of the 3D object.
7. The computer-implemented method of claim 1 , wherein the normalized depth values are computed relative to a reference keypoint.
8. The computer-implemented method of claim 7 , wherein computing the scale normalized 3D pose is based on a depth of the reference keypoint that is calculated using the locations.
9. The computer-implemented method of claim 1 , wherein the normalized depth values are invariant for changes in a scale of the 3D object.
10. The computer-implemented method of claim 1 , wherein the normalized depth values are invariant for changes in a translation of the 3D object.
11. A system, comprising:
a processor configured to:
receive locations of keypoints for a three-dimensional (3D) object, wherein each location includes pixel coordinates and a normalized depth value, the pixel coordinates corresponding to pixels within a two-dimensional (2D) image of the 3D object, the 2D image associated with camera attributes and the normalized depth values corresponding to normalized relative depth values of each one of the keypoints with respect to a reference keypoint;
compute a depth of the reference keypoint with respect to a camera based on the locations and the camera attributes; and
compute a scale normalized 3D pose of the 3D object based on the locations, the depth of the reference keypoint and the camera attributes.
12. The system of claim 11 , wherein the processor is further configured to receive a scale factor corresponding to the 3D object.
13. The system of claim 12 , wherein the processor is further configured to generate, according to the scale factor, an absolute 3D pose of the 3D object from the scale normalized 3D pose.
14. The system of claim 12 , wherein the scale factor is estimated and corresponds to a component of the 3D object.
15. The system of claim 12 , wherein the scale factor is a distance between two vertices or two keypoints of the 3D object.
16. The system of claim 11 , wherein the normalized depth values are computed relative to a reference keypoint.
17. The system of claim 16 , wherein computing the scale normalized 3D pose is based on a depth of the reference keypoint that is calculated using the locations.
18. A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving locations of keypoints for a three-dimensional (3D) object, wherein each location includes pixel coordinates and a normalized depth value, the pixel coordinates corresponding to pixels within a two-dimensional (2D) image of the 3D object, the 2D image associated with camera attributes and the normalized depth values corresponding to normalized relative depth values of each one of the keypoints with respect to a reference keypoint;
computing a depth of the reference keypoint with respect to a camera based on the locations and the camera attributes; and
computing a scale normalized 3D pose of the 3D object based on the locations, the depth of the reference keypoint and the camera attributes.
19. The non-transitory computer-readable media of claim 18 , further comprising receiving a constant scale factor corresponding to the 3D object.
20. The non-transitory computer-readable media of claim 19 , further comprising generating, according to a scale factor, an absolute 3D pose of the 3D object from the scale normalized 3D pose.