Hand pose estimation from stereo cameras
Systems and methods herein describe using a neural network to identify a first set of joint location coordinates and a second set of joint location coordinates and identifying a three-dimensional hand pose based on both the first and second sets of joint location coordinates.
1. A method comprising:
receiving, from a camera, a plurality of images of a hand;
generating a plurality of sets of joint location coordinates by:
for each given image in the plurality of images:
cropping, using one or more processors, a portion of the given image comprising the hand;
identifying, using a neural network, a first set of joint location coordinates in the cropped portion of the given image, wherein the first set of joint location coordinates represent pixel locations of the hand relative to the cropped portion of the given image;
identifying an intermediate set of joint location coordinates, wherein the intermediate set of joint location coordinates represent pixel locations of the hand relative to the given image;
generating a second set of joint location coordinates based on the first set of joint location coordinates and the intermediate set, wherein the second set of joint location coordinates represents joint locations of the hand relative to a three-dimensional physical space; and
identifying a three-dimensional hand pose of the hand based on the plurality of sets of joint location coordinates.
2. The method of claim 1 , wherein the plurality of images comprises a plurality of views of the hand.
3. The method of claim 1 , further comprising:
prompting a user of a client device to initialize a hand position;
receiving the initialized hand position; and
tracking the hand based on the initialized hand position.
4. The method of claim 1 , wherein the camera is a stereo camera.
5. The method of claim 1 ,
wherein the intermediate set of joint location coordinates is measured relative to an uncropped version of the given image.
6. The method of claim 1 , further comprising:
generating a synthetic training dataset comprising stereo image pairs of virtual hands and corresponding ground truth labels, wherein the corresponding ground truth labels comprise joint locations.
7. A system comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the system to perform operations comprising:
receiving, from a camera, a plurality of images of a hand;
generating a plurality of sets of joint location coordinates by:
for each given image in the plurality of images:
cropping, using one or more processors, a portion of the given image comprising the hand;
identifying, using a neural network, a first set of joint location coordinates in the cropped portion of the given image, wherein the first set of joint location coordinates represent pixel locations of the hand relative to the cropped portion of the given image;
identifying an intermediate set of joint location coordinates, wherein the intermediate set of joint location coordinates represent pixel locations of the hand relative to the given image;
generating a second set of joint location coordinates based on the first set of joint location coordinates and the intermediate set, wherein the second set of joint location coordinates represents joint locations of the hand relative to a three-dimensional physical space; and
identifying a three-dimensional hand pose of the hand based on the plurality of sets of joint location coordinates.
8. The system of claim 7 , wherein the plurality of images comprises a plurality of views of the hand.
9. The system of claim 7 , further comprising:
prompting a user of a client device to initialize a hand position;
receiving the initialized hand position; and
tracking the hand based on the initialized hand position.
10. The system of claim 7 , wherein the camera is a stereo camera.
11. The system of claim 7 ,
wherein the intermediate set of joint location coordinates is measured relative to an uncropped version of the given image.
12. The system of claim 7 , further comprising:
generating a synthetic training dataset comprising stereo image pairs of virtual hands and corresponding ground truth labels, wherein the corresponding ground truth labels comprise joint locations.
13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
receiving, from a camera, a plurality of images of a hand;
generating a plurality of sets of joint location coordinates by:
for each given image in the plurality of images:
cropping, using one or more processors, a portion of the given image comprising the hand;
identifying, using a neural network, a first set of joint location coordinates in the cropped portion of the given image, wherein the first set of joint location coordinates represent pixel locations of the hand relative to the cropped portion of the given image;
identifying an intermediate set of joint location coordinates, wherein the intermediate set of joint location coordinates represent pixel locations of the hand relative to the given image;
generating a second set of joint location coordinates based on the first set of joint location coordinates and the intermediate set, wherein the second set of joint location coordinates represents joint locations of the hand relative to a three-dimensional physical space; and
identifying a three-dimensional hand pose of the hand based on the plurality of sets of joint location coordinates.
14. The computer-readable storage medium of claim 13 , wherein the plurality of images comprises a plurality of views of the hand.
15. The computer-readable storage medium of claim 13 , further comprising:
prompting a user of a client device to initialize a hand position;
receiving the initialized hand position; and
tracking the hand based on the initialized hand position.
16. The computer-readable storage medium of claim 13 , wherein the second set of joint location coordinates is measured use millimeters.
17. The computer-readable storage medium of claim 13 , further comprising:
generating a synthetic training dataset comprising stereo image pairs of virtual hands and corresponding ground truth labels, wherein the corresponding ground truth labels comprise joint locations.
18. The computer-readable storage medium of claim 13 ,
wherein the intermediate set of joint location coordinates is measured relative to an uncropped version of the given image.