Machine learning based gesture recognition
The technology disclosed introduces two types of neural networks: “master” or “generalists” networks and “expert” or “specialists” networks. Both, master networks and expert networks, are fully connected neural networks that take a feature vector of an input hand image and produce a prediction of the hand pose. Master networks and expert networks differ from each other based on the data on which they are trained. In particular, master networks are trained on the entire data set. In contrast, expert networks are trained only on a subset of the entire dataset. In regards to the hand poses, master networks are trained on the input image data representing all available hand poses comprising the training data (including both real and simulated hand images).
1. A method of preparing a plurality of neural network systems to recognize hand positions, the method including:
generating a plurality of simulated hand position images, each hand position image labeled with a plurality of labeled hand position parameters, the simulated hand position images organized as gesture sequences;
producing reduced dimensionality images from the simulated hand position images;
training a first set of atemporal generalist neural networks with the simulated hand position images to produce estimated hand position parameters, using the reduced dimensionality images and the labeled hand position parameters for the reduced dimensionality images;
subdividing the simulated hand position images into a plurality of at least partially overlapping specialist categories and training a plurality of corresponding atemporal specialist neural networks to produce estimated hand position parameters;
training a first set of atemporal specialist neural networks using the reduced dimensionality images from the plurality of at least partially overlapping corresponding specialist categories using the reduced dimensionality images from the simulated hand position images and the labeled hand position parameters for the reduced dimensionality images; and
saving parameters from training the first set of atemporal generalist neural networks and the first set of atemporal specialist neural networks in tangible machine readable memory.
2. The method of claim 1 , wherein the simulated hand position images are stereoscopic images.
3. The method of claim 1 , wherein the simulated hand position images further include depth map information.
4. The method of claim 1 , wherein hand position parameters include at least one of a plurality of joint locations in three-dimensional (3D) space.
5. The method of claim 1 , wherein hand position parameters include at least one of a plurality of joint angles in three-dimensional (3D) space.
6. The method of claim 1 , wherein hand position parameters include at least one of a plurality of hand skeleton segments in three-dimensional (3D) space.
7. The method of claim 1 , wherein the at least partially overlapping specialist categories are generated using unsupervised classification.
8. The method of claim 1 , wherein generalist neural networks are trained on an entirety of a dataset comprising of simulated hand position images and specialist neural networks are trained on parts of the dataset corresponding to the plurality of at least partially overlapping specialist categories.
9. The method of claim 1 , further including calculating at least one characterization for each specialist neural network that positions a particular specialist neural network in distinction from other specialist neural networks.
10. The method of claim 1 , further including, during testing:
receiving a first set of hand position parameters from one or more trained generalist neural networks and identifying specialist categories with centroids proximate to centroids in hand positions of the first set of hand position parameters received;
receiving a second set of hand position parameters from a multitude of trained specialist neural networks corresponding to the specialist categories identified; and
combining the first set of hand position parameters and the second set of hand position parameters to generate a final hand pose estimate.
11. The method of claim 10 , further including combining the first set of hand position parameters and the second set of hand position parameters to generate a final hand pose estimate using an outlier-robust covariance propagation scheme.
12. The method of claim 1 , wherein each generalist neural network and each specialist neural network generates 84 outputs representing 28 hand joint locations in three-dimensional (3D) space.
13. A method of preparing a plurality of neural network systems to recognize hand positions, the method including:
generating a plurality of simulated hand position images, each hand position image labeled with a plurality of labeled hand position parameters, the simulated hand position images organized as gesture sequences;
producing reduced dimensionality images from the simulated hand position images;
training a first set of temporal generalist neural networks with the simulated hand position images to produce estimated hand position parameters, using pairs of first and second reduced dimensionality images, estimated or actual hand position parameters for the first reduced dimensionality image, image data for the second reduced dimensionality image, and the labeled hand position parameters for the second reduced dimensionality image;
subdividing the simulated hand position images into a plurality of at least partially overlapping specialist categories and training a plurality of corresponding temporal specialist neural networks to produce estimated hand position parameters;
training a first set of temporal specialist neural networks using pairs of first and second reduced dimensionality images from the plurality of at least partially overlapping corresponding specialist categories, including:
estimated or actual hand position parameters for the first reduced dimensionality image,
image data for the second reduced dimensionality image, and
the labeled hand position parameters for the second reduced dimensionality image; and
saving parameters from training the first set of temporal generalist neural networks and the first set of temporal specialist neural networks in tangible machine readable memory.
14. The method of claim 13 , wherein the first set of temporal generalist neural networks and the first set of temporal specialist neural networks include recursive neural networks (RNN) based on long short term memory (LSTM).
15. The method of claim 13 , wherein the first set of temporal generalist neural networks and the first set of temporal specialist neural networks are trained using a combination of current simulated hand position images and additional noise hand position data.
16. The method of claim 13 , wherein the first set of temporal generalist neural networks and the first set of temporal specialist neural networks are trained using a series of simulated hand position images that are temporally linked as gesture sequences representing real world hand gestures.
17. The method of claim 13 , wherein the first set of temporal generalist neural networks and the first set of temporal specialist neural networks, during testing, utilize a combination of a current simulated hand position image and a series of prior estimated hand position parameters temporally linked in previous frames to generate a current set of hand position parameters.
18. A non-transitory computer readable storage medium impressed with computer program instructions, which instructions, when executed on a processor, implement a method including actions of:
generating a plurality of simulated hand position images, each hand position image labeled with a plurality of labeled hand position parameters, the simulated hand position images organized as gesture sequences;
producing reduced dimensionality images from the simulated hand position images;
training a first set of atemporal generalist neural networks with the simulated hand position images to produce estimated hand position parameters, using the reduced dimensionality images and the labeled hand position parameters for the reduced dimensionality images;
subdividing the simulated hand position images into a plurality of overlapping specialist categories and training a plurality of corresponding atemporal specialist neural networks to produce estimated hand position parameters;
training a first set of atemporal specialist neural networks using the reduced dimensionality images from the plurality of overlapping corresponding specialist categories using the reduced dimensionality images from the simulated hand position images and the labeled hand position parameters for the reduced dimensionality images; and
saving parameters from training the first set of atemporal generalist neural networks and the first set of atemporal specialist neural networks in tangible machine readable memory.
19. A non-transitory computer readable storage medium impressed with computer program instructions, which instructions, when executed on a processor, implement a method including actions of:
generating a plurality of simulated hand position images, each hand position image labeled with a plurality of labeled hand position parameters, the simulated hand position images organized as gesture sequences;
producing reduced dimensionality images from the simulated hand position images;
training a first set of temporal generalist neural networks with the simulated hand position images to produce estimated hand position parameters, using pairs of first and second reduced dimensionality images, estimated or actual hand position parameters for the first reduced dimensionality image, image data for the second reduced dimensionality image, and the labeled hand position parameters for the second reduced dimensionality image;
subdividing the simulated hand position images into a plurality of at least partially overlapping specialist categories and training a plurality of corresponding temporal specialist neural networks to produce estimated hand position parameters;
training a first set of temporal specialist neural networks using pairs of first and second reduced dimensionality images from the plurality of at least partially overlapping corresponding specialist categories, including:
estimated or actual hand position parameters for the first reduced dimensionality image,
image data for the second reduced dimensionality image, and
the labeled hand position parameters for the second reduced dimensionality image; and
saving parameters from training the first set of temporal generalist neural networks and the first set of temporal specialist neural networks in tangible machine readable memory.
20. A system comprising a computer readable storage medium coupled with a processor, the computer readable storage medium storing computer program instructions, which instructions, when executed on a processor, implement a method of claim 1 .
21. A system comprising a computer readable storage medium coupled with a processor, the computer readable storage medium storing computer program instructions, which instructions, when executed on a processor, implement a method of claim 13 .