Method and system for gesture recognition
View Patent ↗A method of image acquisition and data pre-processing includes obtaining from a sensor an image of a subject making a movement. The sensor may be a depth camera. The method also includes selecting a plurality of features of interest from the image, sampling a plurality of depth values corresponding to the plurality of features of interest, projecting the plurality of features of interest onto a model utilizing the plurality of depth values, and constraining the projecting of the plurality of features of interest onto the model utilizing a constraint system. The constraint system may comprise an inverse kinematics solver.
1. A method of recognizing a gesture of interest comprising:
prompting a subject to perform the gesture of interest, wherein a sequence of baseline depth images with three-dimensional baseline positions of feature points are associated with the gesture of interest;
obtaining from a depth sensor a plurality of depth images of the subject making movements;
identifying a first set of three-dimensional positions of a plurality of feature points in each of the plurality of depth images;
projecting the first set of three-dimensional positions of the plurality of feature points onto a constrained three-dimensional model for each of the plurality of depth images;
mapping the first set of three-dimensional positions of the plurality of features using the constrained model for each of the plurality of depth images independently of the other plurality of depth images;
determining whether the mapped first set of three-dimensional positions of the feature points are quantitatively similar to the three-dimensional baseline positions of feature points in the one or more baseline depth images of a pre-determined gesture;
independently comparing the mapped first set of three-dimensional positions of the plurality of feature points for each of the plurality of depth images to the three-dimensional baseline positions of feature points in the sequence of baseline depth images for the gesture of interest as each of the plurality of depth images is obtained;
determining a tracking score based on the comparing; and
determining that the subject is performing the gesture of interest if the tracking score remains within a given threshold.
2. The method of claim 1 , further comprising selecting the gesture of interest from a gesture library.
3. The method of claim 1 , wherein the comparing includes computing a similarity measure.
4. The method of claim 1 , wherein the depth sensor comprises a depth camera.
5. The method of claim 1 , wherein the constrained model comprises an inverse kinematics solver.
6. The method of claim 1 , further comprising animation retargeting for scaling the constrained model onto a standard model.
7. A system for recognizing gestures, comprising:
a depth sensor for acquiring multiple frames of image depth data;
an image acquisition module configured to receive the multiple frames of image depth data from the depth sensor and process the multiple frames of image depth data wherein processing comprises:
identifying three dimensional positions of feature points in each of the multiple frames of image depth data;
projecting the three dimensional positions of feature points onto a constrained three-dimensional model for each of the multiple frames of image depth data;
mapping the three-dimensional positions of the feature points using the constrained model for each of the multiple frames of image depth data independently of the other multiple frames;
a library of pre-determined gestures, wherein each pre-determined gesture is associated with one or more baseline depth images having three-dimensional baseline positions of feature points;
a binary gesture recognition module configured to receive the mapped three-dimensional positions of the feature points of the subject from the image acquisition module and determine whether the mapped three-dimensional positions of the feature points are quantitatively similar to the three-dimensional baseline positions of feature points in the one or more baseline depth images of a pre-determined gesture in the library;
a real-time gesture recognition module configured to receive the mapped three-dimensional positions of the feature points of the subject from the image acquisition module, compare the mapped three-dimensional positions of the feature points for each of the multiple frames of image depth data to the three-dimensional baseline positions of feature points in the one or more baseline depth images associated with a prompted gesture of interest as each of the plurality of depth images is obtained to determine a tracking score and determine that the subject is performing the gesture of interest if the tracking score remains within a given threshold.
8. The system of claim 7 , further comprising:
a game engine module configured to select the prompted gesture and prompt the subject to perform the prompted gesture.
9. The system of claim 7 , further comprising:
a display for providing feedback to the subject about gestures performed by the subject.
10. The system of claim 7 wherein the camera further acquires color image data.
11. The system of claim 7 wherein the real-time gesture recognition module is further configured to calculate a similarity measure and a cumulative tracking score, wherein the similarity measure and the cumulative tracking score are updated for each frame independently of the other frames, and further wherein the determination whether the particular gesture is being performed is based upon comparing the cumulative tracking score to a threshold for the particular gesture for each frame.
12. The system of claim 7 wherein the constrained model comprises an inverse kinematic solver.
13. The system of claim 7 wherein the image acquisition module scales the feature positions to a standard model.
14. The system of claim 7 wherein the gesture training module uses machine learning techniques to determine whether the feature positions match a particular gesture.