Hand initialization for machine learning based gesture recognition
The technology disclosed also initializes a new hand that enters the field of view of a gesture recognition system using a parallax detection module. The parallax detection module determines candidate regions of interest (ROI) for a given input hand image and computes depth, rotation and position information for the candidate ROI. Then, for each of the candidate ROI, an ImagePatch, which includes the hand, is extracted from the original input hand image to minimize processing of low-information pixels. Further, a hand classifier neural network is used to determine which ImagePatch most resembles a hand. For the qualified, most-hand like ImagePatch, a 3D virtual hand is initialized with depth, rotation and position matching that of the qualified ImagePatch.
1. A method of determining a hand pose using neural network systems, the method including:
receiving a first set of estimates of hand position parameters from multiple generalist neural networks and/or specialist neural networks for each of a plurality of hand joints;
for each individual hand joint, simultaneously determining a principal distribution of the first set of estimates;
receiving a second set of estimates of hand position parameters from the generalist neural networks and/or specialist neural networks for each of the plurality of hand joints; and
for each individual hand joint, simultaneously:
calculating a similarity measure between the second set of estimates and the principal distribution of the first set of estimates;
identifying select estimates in the second set of estimates based on the similarity measure;
calculating contribution weights of the select estimates based on the similarity measure; and
determining a principal distribution of the second set of estimates based on the contribution weights of the select estimates.
2. The method of claim 1 , further including identifying outliers and inliers as select estimates.
3. The method of claim 1 , further including determining a hand pose by minimizing an approximation error between principal distributions of each of the hand joints.
4. The method of claim 1 , wherein the hand position parameters of at least one of the first set of estimates and the second set of estimates are a plurality of joint locations in three-dimensional (3D) space.
5. The method of claim 1 , wherein the hand position parameters of at least one of the first set of estimates and the second set of estimates are a plurality of joint angles in three-dimensional (3D) space.
6. The method of claim 1 , wherein the hand position parameters of at least one of the first set of estimates and the second set of estimates are a plurality of hand skeleton segments in three-dimensional (3D) space.
7. The method of claim 1 , wherein the principal distribution is a covariance matrix of the hand position parameter estimates.
8. The method of claim 1 , wherein the similarity measure is a Mahalanobis distance from the principal distribution.
9. The method of claim 1 , wherein the similarity measure is a projection statistic from the principal distribution.
10. The method of claim 7 , wherein the covariance matrix is determined using a Kalman filter operation.
11. The method of claim 7 , wherein the covariance matrix is updated between frames based on contribution weights of outliers and inliers of a current set of estimates of hand position parameters.
12. The method of claim 1 , wherein the contribution weights are determined by converting the similarity measure into probability distributions.
13. A non-transitory computer readable storage medium impressed with computer program instructions to determine a hand pose using neural network systems, the instructions, when executed on a processor, implement a method comprising:
receiving a first set of estimates of hand position parameters from multiple generalist neural networks and/or specialist neural networks for each of a plurality of hand joints;
for each individual hand joint, simultaneously determining a principal distribution of the first set of estimates;
receiving a second set of estimates of hand position parameters from the generalist neural networks and/or specialist neural networks for each of the plurality of hand j oints; and
for each individual hand joint, simultaneously:
calculating a similarity measure between the second set of estimates and the principal distribution of the first set of estimates;
identifying outliers and inliers in the second set of estimates based on the similarity measure;
calculating contribution weights of the outliers and the inliers based on the similarity measure; and
determining a principal distribution of the second set of estimates based on the contribution weights of the outliers and inliers.
14. The non-transitory computer readable storage medium of claim 13 , implementing the method further comprising determining a hand pose by minimizing an approximation error between principal distributions of each of the hand joints.
15. The non-transitory computer readable storage medium of claim 13 , wherein the hand position parameters of at least one of the first set of estimates and the second set of estimates are a plurality ofjoint locations in three-dimensional (3D) space.
16. The non-transitory computer readable storage medium of claim 13 , wherein the hand position parameters of at least one of the first set of estimates and the second set of estimates are a plurality ofjoint angles in three-dimensional (3D) space.
17. The non-transitory computer readable storage medium of claim 13 , wherein the hand position parameters of at least one of the first set of estimates and the second set of estimates are a plurality of hand skeleton segments in three-dimensional (3D) space.
18. The non-transitory computer readable storage medium of claim 13 , wherein the principal distribution is a covariance matrix of the hand position parameter estimates.
19. The non-transitory computer readable storage medium of claim 13 , wherein the similarity measure is a Mahalanobis distance from the principal distribution.
20. The non-transitory computer readable storage medium of claim 13 , wherein the contribution weights are determined by converting the similarity measure into probability distributions.
21. A system including one or more processors coupled to memory, the memory loaded with computer instructions to determine a hand pose using neural network systems, the instructions, when executed on the processors, implement actions comprising:
receiving a first set of estimates of hand position parameters from multiple generalist neural networks and/or specialist neural networks for each of a plurality of hand joints;
for each individual hand joint, simultaneously determining a principal distribution of the first set of estimates;
receiving a second set of estimates of hand position parameters from the generalist neural networks and/or specialist neural networks for each of the plurality of hand joints; and
for each individual hand joint, simultaneously:
calculating a similarity measure between the second set of estimates and the principal distribution of the first set of estimates;
identifying outliers and inliers in the second set of estimates based on the similarity measure;
calculating contribution weights of the outliers and the inliers based on the similarity measure; and
determining a principal distribution of the second set of estimates based on the contribution weights of the outliers and inliers.
22. A method of determining a hand pose using neural network systems, the method including:
receiving (i) a first set of estimates of hand position parameters from multiple generalist neural networks and/or specialist neural networks for each of a plurality of hand joints; (ii) a principal distribution of the first set of estimates determined for each individual hand joint; and (iii) a second set of estimates of hand position parameters from the generalist neural networks and/or specialist neural networks for each of the plurality of hand joints; and
for each individual hand joint:
calculating a similarity measure between the second set of estimates and the principal distribution of the first set of estimates;
identifying select estimates in the second set of estimates based on the similarity measure;
calculating contribution weights of the select estimates based on the similarity measure; and
determining a principal distribution of the second set of estimates based on the contribution weights of the select estimates.
23. A method of determining a hand pose using neural network systems, the method including:
receiving a principal distribution of a second set of estimates of hand position parameters having been determined based on contribution weights calculated of select estimates identified in the second set of estimates based on a calculated similarity measure calculated for individual ones of a plurality of hand joints between the second set of estimates and a principal distribution of a first set of estimates; wherein the (i) a first set of estimates of hand position parameters for individual ones of a plurality of hand joints and the (ii) the second set of estimates of hand position parameters for individual ones of a plurality of hand joints are obtained from multiple generalist neural networks and/or specialist neural networks, and
using the received principle distribution to inform a hand pose based upon the first set of estimates of hand position parameters and the second set of hand position parameters.
24. A system including one or more processors coupled to memory, the memory loaded with computer instructions to determine a hand pose using neural network systems, the instructions, when executed on the processors, implement actions comprising:
obtaining a principal distribution of a second set of estimates of hand position parameters having been determined based on contribution weights calculated of select estimates identified in the second set of estimates based on a calculated similarity measure calculated for individual ones of a plurality of hand joints between the second set of estimates and a principal distribution of a first set of estimates; wherein the (i) a first set of estimates of hand position parameters for individual ones of a plurality of hand joints and the (ii) the second set of estimates of hand position parameters for individual ones of a plurality of hand joints are obtained from multiple generalist neural networks and/or specialist neural networks, and
using the received principle distribution to inform a hand pose based upon the first set of estimates of hand position parameters and the second set of hand position parameters.