Hand calibration using single depth camera
A system generates a user hand shape model from a single depth camera. The system includes the single depth camera and a hand tracking unit. The single depth camera generates single depth image data of a user's hand. The hand tracking unit applies the single depth image data to a neural network model to generate heat maps indicating locations of hand features. The locations of hand features are used to generate a user hand shape model customized to the size and shape of the user's hand. The user hand shape model is defined by a set of principal component hand shapes defining a hand shape variation space. The limited number of principal component hand shape models reduces determination of user hand shape to a smaller number of variables, and thus provides for a fast calibration of the user hand shape model.
1. A system comprising:
a camera configured to generate image data including multiple views of a user's hand;
a circuitry configured to:
generate input hand shape models of the user's hand using the image data, each input hand shape model defining a hand size of the user determined from one of the multiple views;
determine a user hand shape model by fitting the input hand shape models to a set of principal component hand shape models, the principal component hand shape models defining orthogonal dimensions of hand size variance with respect to each other such that the set of principal component hand shape models defines a hand size variation space; and
render an image of the user's hand using the user hand shape model; and
a display device configured to present the rendered image to the user.
2. The system of claim 1 , wherein:
the camera is a single depth camera; and
the image data includes depth image data.
3. The system of claim 1 , further comprising a plurality of cameras at different locations, the plurality of cameras configured to generate sets of the image data, each set of the image data including multiple views of a pose of the user's hand, and wherein the input hand shape models are generated using the sets of the image data.
4. The system of claim 1 , wherein the circuitry configured to generate the input hand shape models using the image data includes the circuitry being configured to apply the image data to a neural network model.
5. The system of claim 1 , wherein the user hand shape model is defined by a set of coefficients defining a weighted combination of the set of principal component hand shape models.
6. The system of claim 1 , wherein the circuitry configured to determine the user hand shape model by fitting the input hand shape models to the set of principal component hand shape models includes the circuitry being configured to:
map vertices of an input hand shape model to vertices of principal hand meshes of the set of principal component hand shape models;
determine error values between the vertices of the input hand shape model and the vertices of the principal hand meshes; and
determine the user hand shape model as a combination of the principal component hand shape models that minimizes the error values between the vertices of the input hand shape model and the vertices of the principal hand meshes.
7. The system of claim 1 , wherein the circuitry is further configured to:
receive second image data of the user's hand from the camera;
determine, based on applying the second image data to a neural network model, heat maps indicating locations of hand features; and
determine a pose of the user's hand based on fitting the locations of the hand features to the user hand shape model.
8. The system of claim 1 , wherein the camera and the display device are components of a head-mounted display.
9. A method, comprising, by a circuitry:
generating input hand shape models of a user's hand using image data generated by a camera, the image data including multiple views of the user's hand, each input hand shape model defining a hand size of the user determined from one of the multiple views;
determining a user hand shape model by fitting the input hand shape models to a set of principal component hand shape models, the principal component hand shape models defining orthogonal dimensions of hand size variance with respect to each other such that the set of principal component hand shape models defines a hand size variation space;
rendering an image of the user's hand using the user hand shape model; and
providing the image of the user's hand to a display device.
10. The method of claim 9 , wherein:
the camera is a single depth camera; and
the image data includes depth image data.
11. The method of claim 9 , wherein the input hand shape models are generated using the sets of the image data, each set of the image data including multiple views of a pose of the user's hand generated by a plurality of cameras at different locations.
12. The method of claim 9 , wherein generating the input hand shape models using the image data includes applying the image data to a neural network model.
13. The method of claim 9 , wherein the user hand shape model is defined by a set of coefficients defining a weighted combination of the set of principal component hand shape models.
14. The method of claim 9 , wherein determining the user hand shape model by fitting the input hand shape models to the set of principal component hand shape models includes:
mapping vertices of an input hand shape model to vertices of principal hand meshes of the set of principal component hand shape models;
determining error values between the vertices of the input hand shape model and the vertices of the principal hand meshes; and
determining the user hand shape model as a combination of the principal component hand shape models that minimizes the error values between the vertices of the input hand shape model and the vertices of the principal hand meshes.
15. The method of claim 9 , further comprising, by the circuitry:
receiving second image data of the user's hand from the camera;
determining, based on applying the second image data to a neural network model, heat maps indicating locations of hand features; and
determining a pose of the user's hand based on fitting the locations of the hand features to the user hand shape model.
16. The method of claim 9 , wherein the camera and the display device are components of a head-mounted display.
17. A non-transitory computer readable medium comprising stored instructions that, when executed by at least one processor, configure the at least one processor to:
generate input hand shape models of the user's hand using the image data, each input hand shape model defining a hand size of the user determined from one of the multiple views;
determine a user hand shape model by fitting the input hand shape models to a set of principal component hand shape models, the principal component hand shape models defining orthogonal dimensions of hand size variance with respect to each other such that the set of principal component hand shape models defines a hand size variation space;
render an image of the user's hand using the user hand shape model; and
provide the image of the user's hand to a display device.
18. The non-transitory computer readable medium of claim 17 , wherein the stored instructions that configure the at least one processor to generate the input hand shape models using the image data further comprises stored instructions that configure the at least one processor to apply the image data to a neural network model.
19. The non-transitory computer readable medium of claim 17 , wherein the user hand shape model is defined by a set of coefficients defining a weighted combination of the set of principal component hand shape models.
20. The non-transitory computer readable medium of claim 17 , wherein the stored instructions further configure the at least one processor to:
receive second image data of the user's hand from the camera;
determine, based on applying the second image data to a neural network model, heat maps indicating locations of hand features; and
determine a pose of the user's hand based on fitting the locations of the hand features to the user hand shape model.