Method and system for hand pose detection
A method for hand pose identification in an automated system includes providing depth map data of a hand of a user to a first neural network trained to classify features corresponding to a joint angle of a wrist in the hand to generate a first plurality of activation features and performing a first search in a predetermined plurality of activation features stored in a database in the memory to identify a first plurality of hand pose parameters for the wrist associated with predetermined activation features in the database that are nearest neighbors to the first plurality of activation features. The method further includes generating a hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters and performing an operation in the automated system in response to input from the user based on the hand pose model.
1. A method for identification of a hand pose as input to an automated system comprising:
providing, with a processor in the automated system, depth map data of a hand of a user to a first neural network trained to classify features corresponding to a joint angle of a wrist in the hand to generate a first plurality of activation features;
performing, with the processor and a recommendation engine stored in the memory, a first search in a predetermined plurality of activation features stored in a database in the memory to identify a first plurality of hand pose parameters for the wrist associated with predetermined activation features in the database that are nearest neighbors to the first plurality of activation features;
generating, with the processor, a hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters; and
performing, with the processor in the automated system, an operation in response to input from the user based at least in part on the hand pose model.
2. The method of claim 1 further comprising:
identifying, with the processor, a second neural network stored in the memory based upon the first plurality of hand pose parameters, the second neural network being one neural network in a plurality of neural networks stored in the memory trained to classify features corresponding to joint angles of a first finger of the hand;
providing, with the processor, the depth map data of the hand of the user to the second neural network to generate a second plurality of activation features;
performing, with the processor and the recommendation engine, a second search in the predetermined plurality of activation features stored in the database to identify a second plurality of hand pose parameters for the one finger associated with predetermined activation features in the database that are nearest neighbors to the second plurality of activation features; and
generating, with the processor, the hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters and the second plurality of hand pose parameters.
3. The method of claim 2 further comprising:
identifying, with the processor, a plurality of neural networks stored in the memory based upon the first plurality of hand pose parameters, each neural network in the plurality of neural networks being trained to classify features corresponding to joint angles of one of a second finger, a third finger, a fourth finger, and a fifth finger of the hand;
providing, with the processor, the depth map data of the hand of the user to each neural network in the plurality of neural networks to generate a plurality of activation features for each of the plurality of neural networks, each plurality of activation features corresponding to joint angles of one of the second, third, fourth, and fifth fingers in the depth map data;
performing, with the processor and the recommendation engine, a plurality of searches in the predetermined plurality of activation features stored in the database based on the plurality of activation features for each of the second, the third, the fourth, and the fifth fingers to identify additional pluralities of hand pose parameters for the second, the third, the fourth, and the fifth fingers; and
generating, with the processor, the hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters, the second plurality of hand pose parameters, and the additional pluralities of hand pose parameters for the second, the third, the fourth, and the fifth fingers.
4. The method of claim 2 , the identifying of the second neural network further comprising:
identifying, with the processor, a predetermined subset of a range of joint angles for the wrist that includes the joint angle of the wrist in the first plurality of hand pose parameters; and
identifying, with the processor, the second neural network based on a predetermined relationship between the predetermined subset of the range and the plurality of neural networks stored in the memory.
5. The method of claim 1 , the first search further comprising:
a spatial search of the predetermined plurality of activation features in the database to identify the first plurality of hand pose parameters.
6. The method of claim 1 , the first search further comprising:
a spatial-temporal search of another plurality of activation features and associated hand pose parameters in the database corresponding to at least one previous depth map received within a predetermined time period of generation of the depth map data to identify the first plurality of hand pose parameters.
7. A system for computer human interaction comprising:
a depth camera configured to generate depth map data of a hand of a user;
an output device;
a memory; and
a processor operatively connected to the depth camera, the output device, and the memory, the processor being configured to:
receive depth map data of a hand of a user from the depth camera;
provide the depth map data to a first neural network stored in the memory, the first neural network being trained to classify features corresponding to a joint angle of a wrist in the hand to generate a first plurality of activation features;
perform a first search, using a recommendation engine stored in the memory, in a predetermined plurality of activation features stored in a database stored in the memory to identify a first plurality of hand pose parameters for the wrist associated with predetermined activation features in the database that are nearest neighbors to the first plurality of activation features;
generate a hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters; and
generate an output with the output device in response to input from the user based at least in part on the hand pose model.
8. The system of claim 7 , the processor being further configured to:
identify a second neural network stored in the memory based upon the first plurality of hand pose parameters, the second neural network being one neural network in a plurality of neural networks stored in the memory trained to classify features corresponding to joint angles of a first finger of the hand;
provide the depth map data of the hand of the user to the second neural network to generate a second plurality of activation features;
perform a second search, using the recommendation engine, in the predetermined plurality of activation features stored in the database to identify a second plurality of hand pose parameters for the one finger associated with predetermined activation features in the database that are nearest neighbors to the second plurality of activation features; and
generate the hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters and the second plurality of hand pose parameters.
9. The system of claim 8 , the processor being further configured to:
identify a plurality of neural networks stored in the memory based upon the first plurality of hand pose parameters, each neural network in the plurality of neural networks being trained to classify features corresponding to joint angles of one of a second finger, a third finger, a fourth finger, and a fifth finger of the hand;
provide the depth map data of the hand of the user to each neural network in the plurality of neural networks to generate a plurality of activation features for each of the plurality of neural networks, each plurality of activation features corresponding to joint angles of one of the second, third, fourth, and fifth fingers in the depth map data;
perform a plurality of searches, using the recommendation engine, in the predetermined plurality of activation features stored in the database based on the plurality of activation features for each of the second, the third, the fourth, and the fifth fingers to identify additional pluralities of hand pose parameters for the second, the third, the fourth, and the fifth fingers; and
generate the hand pose model corresponding to the hand of the user based on the first plurality of hand pose parameters, the second plurality of hand pose parameters, and the additional pluralities of hand pose parameters for the second, the third, the fourth, and the fifth fingers.
10. The system of claim 8 , the processor being further configured to:
identify a predetermined subset of a range of joint angles for the wrist that includes the joint angle of the wrist in the first plurality of hand pose parameters; and
identify the second neural network based on a predetermined relationship between the predetermined subset of the range and the plurality of neural networks stored in the memory.
11. The system of claim 7 , the processor being further configured to:
perform the first search as a spatial search, using the recommendation engine, of the predetermined plurality of activation features in the database to identify the first plurality of hand pose parameters.
12. The system of claim 7 , the processor being further configured to:
perform the first search as a spatial-temporal search, using the recommendation engine, of another plurality of activation features and associated hand pose parameters in the database corresponding to at least one previous depth map generated by the depth camera within a predetermined time period of generation of the depth map data to identify the first plurality of hand pose parameters.