Three-dimensional hand tracking using depth sequences
In the field of Human-computer interaction (HCI), i.e., the study of the interfaces between people (i.e., users) and computers, understanding the intentions and desires of how the user wishes to interact with the computer is a very important problem. The ability to understand human gestures, and, in particular, hand gestures, as they relate to HCI, is a very important aspect in understanding the intentions and desires of the user in a wide variety of applications. In this disclosure, a novel system and method for three-dimensional hand tracking using depth sequences is described. Some of the major contributions of the hand tracking system described herein include: 1.) a robust hand detector that is invariant to scene background changes; 2.) a bi-directional tracking algorithm that prevents detected hands from always drifting closer to the front of the scene (i.e., forward along the z-axis of the scene); and 3.) various hand verification heuristics.
1. An apparatus, comprising:
a depth-sensing camera;
a memory having, stored therein, computer program code; and
one or more processing units operatively coupled to the memory and configured to execute instructions in the computer program code that cause the one or more processing units to:
receive a depth map of a scene containing one or more human hands from the depth-sensing camera, the depth map comprising a matrix of pixels, each pixel having a depth value;
extract, from the depth map, features based on the depth values of the pixels in a plurality of patches distributed in respective positions over the one or more human hands, wherein the depth values of the pixels are normalized, such that the features are background-invariant;
match the extracted features to previously-stored features;
estimate a position of at least one of the one or more human hands based, at least in part, on stored information associated with the matched features; and
track the position of the at least one of the one or more human hands,
wherein the instructions to track comprise instructions to track bi-directionally along a z-axis of the scene.
2. The apparatus of claim 1 , wherein the features use a constant value for the depth values of background pixels.
3. The apparatus of claim 1 , wherein the instructions to match the extracted features to previously-stored features further comprise instructions to use a background-invariant decision forest.
4. The apparatus of claim 1 , wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard pixels that do not exhibit a threshold amount of motion.
5. The apparatus of claim 1 , wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard hands that do not exhibit single-directional connectivity to a human body.
6. The apparatus of claim 1 , wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to locate local extrema in the depth map of the scene.
7. The apparatus of claim 1 , wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to weight the x-coordinate value and y-value coordinate of the pixels in the plurality of patches.
8. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to:
receive a depth map of a scene containing one or more human hands from a depth-sensing camera, the depth map comprising a matrix of pixels, each pixel having a depth value;
extract, from the depth map, features based on the depth values of the pixels in a plurality of patches distributed in respective positions over the one or more human hands, wherein the depth values of the pixels are normalized, such that the features are background-invariant;
match the extracted features to previously-stored features;
estimate a position of at least one of the one or more human hands based, at least in part, on stored information associated with the matched features; and
track the position of the at least one of the one or more human hands,
wherein the instructions to track comprise instructions to track bi-directionally along a z-axis of the scene.
9. The non-transitory program storage device of claim 8 , wherein the features use a constant value for the depth values of background pixels.
10. The non-transitory program storage device of claim 8 , wherein the instructions to match the extracted features to previously-stored features further comprise instructions to use a background-invariant decision forest.
11. The non-transitory program storage device of claim 8 , wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard pixels that do not exhibit a threshold amount of motion.
12. The non-transitory program storage device of claim 8 , wherein the instructions to estimate the position of the at least one of the one or more human hands further comprise instructions to disregard hands that do not exhibit single-directional connectivity to a human body.
13. The non-transitory program storage device of claim 8 , wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to locate local extrema in the depth map of the scene.
14. The non-transitory program storage device of claim 8 , wherein the instructions to track bi-directionally along a z-axis of the scene further comprise instructions to weight the x-coordinate value and y-value coordinate of the pixels in the plurality of patches.
15. A computer-implemented method, comprising:
receiving a depth map of a scene containing one or more human hands from a depth-sensing camera, the depth map comprising a matrix of pixels, each pixel having a depth value;
extracting, from the depth map, features based on the depth values of the pixels in a plurality of patches distributed in respective positions over the one or more human hands, wherein the depth values of the pixels are normalized, such that the descriptors are background-invariant;
matching the extracted features to previously-stored features;
estimating a position of at least one of the one or more human hands based, at least in part, on stored information associated with the matched features; and
tracking the position of the at least one of the one or more human hands,
wherein the instructions to track comprise instructions to track bi-directionally along a z-axis of the scene.
16. The computer-implemented method of claim 15 , wherein the features use a constant value for the depth values of background pixels.
17. The computer-implemented method of claim 15 , wherein estimating the position of the at least one of the one or more human hands further comprises disregarding background pixels and pixels that do not exhibit a threshold amount of motion.
18. The computer-implemented method of claim 15 , wherein estimating the position of the at least one of the one or more human hands further comprises disregarding hands that do not exhibit single-directional connectivity to a human body.
19. The computer-implemented method of claim 15 , wherein tracking bi-directionally along a z-axis of the scene further comprises locating local extrema in the depth map of the scene.
20. The computer-implemented method of claim 15 , wherein tracking bi-directionally along a z-axis of the scene further comprises weighting the x-coordinate value and y-value coordinate of the pixels in the plurality of patches.