Gesture recognition
A gesture recognition method comprises receiving at a processor from a sensor a sequence of captured signal frames for extracting hand pose information for a hand and using at least one trained predictor executed on the processor to extract hand pose information from the received signal frames. For at least one defined gesture, defined as a time sequence comprising hand poses, with each of the hand poses defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand, truth values are computed for the qualitative propositions using the hand pose information extracted from the received signal frames, and execution of the gesture is tracked, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next.
1. A gesture recognition method comprising:
receiving at a processor from a sensor a sequence of captured signal frames for extracting hand pose information for a hand;
using at least one trained predictor executed on the processor to extract the hand pose information from the received signal frames;
for at least one defined gesture, wherein the gesture is defined as a time sequence comprising hand poses, wherein each of the hand poses is defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand:
computing truth values for the qualitative propositions using the hand pose information extracted from the received signal frames, and
tracking execution of the gesture, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next, wherein upon completion of the sequence, a function associated with the gesture is triggered.
2. A gesture recognition method according to claim 1 , wherein the interest points comprise the fingertips and palm center of the hand.
3. A gesture recognition method according to claim 1 , wherein the qualitative propositions are obtained by applying predicates to the interest points of the hand.
4. A gesture recognition method according to claim 3 , wherein the predicates comprise at least one of the following predicates: palm direction, palm orientation, finger direction, finger flexion, finger tangency, finger relative position.
5. A gesture recognition method according to claim 1 , wherein the hand pose information is extracted in multiple stages, wherein at each of the stages a piece of the hand pose information is extracted using a set of one or more predictors.
6. A gesture recognition method according to claim 5 , wherein the piece of hand pose information is extracted from each of the signal frames at a later one of the stages in dependence on the piece of hand pose information extracted from that signal frame at an earlier one of the stages.
7. A gesture recognition method according to claim 6 , wherein only a subset of one or more predictors selected from a set of available trained predictors is activated at the later stage to extract the piece of hand pose information from that signal frame at the later stage, that subset of predictors being selected for that signal frame based on the piece of information extracted from that signal frame at the earlier stage.
8. A gesture recognition method according to claim 7 , wherein at the earlier stage a trained classifier is used to classify the signal frame as belonging to at least one of a plurality of global hand orientation clusters, the piece of hand pose information being extracted at the later stage in dependence on the global hand orientation cluster to which the signal frame belongs.
9. A gesture recognition method according to claim 8 , wherein at the later stage the piece of hand pose information is extracted using at least one trained regressor.
10. A gesture recognition method according to claim 9 , wherein only a subset of one or more regressors selected from a set of available trained regressors is activated at the later stage to extract the piece of hand pose information from that signal frame at the later stage, that subset of regressors being selected for that signal frame based on the global hand orientation cluster to which the signal frame belongs.
11. A gesture recognition method according to claim 10 , wherein the subset of one or more regressors is used to determine a refined global hand orientation estimate, or location information for at least one finger of the hand.
12. A gesture recognition method according to claim 11 , wherein regression is performed separately for multiple fingers to determine location information for each of those fingers.
13. A gesture recognition method according to claim 9 , wherein a fixed number of regression stages is performed for each of the signal frames.
14. A gesture recognition method according to claim 1 , wherein the predictors are convolutional tables ensemble (CTE) predictors.
15. A gesture recognition method according to claim 1 , wherein extracting the hand pose information comprises estimating a hand pose for each of the signal frames.
16. A gesture recognition method according to claim 1 , wherein the signal frames are filtered signal frames generated by applying a temporal filter to unfiltered frames captured by the sensor, whereby each of the filtered signal frames comprises information from multiple unfiltered frames.
17. A method according to claim 1 , wherein the sensor comprises a camera and the signal frames are images of the hand captured by the camera.
18. A gesture recognition method according to claim 1 , wherein the time sequence also comprises at least one motion element, which is a qualitative indicator of hand motion.
19. A storage device storing executable instructions that, when executed on a processor, cause the processor to implement a method comprising:
receiving at the processor from a sensor a sequence of captured signal frames for extracting hand pose information for a hand;
using at least one trained predictor executed on the processor to extract hand pose information from the received signal frames;
for at least one defined gesture, wherein the gesture is defined as a time sequence comprising hand poses, wherein each of the hand poses is defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand:
computing truth values for the qualitative propositions using the hand pose information extracted from the received signal frames, and
tracking execution of the gesture, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next, wherein upon completion of the sequence, a function associated with the gesture is triggered.
20. A gesture recognition device comprising:
a sensor for use in capturing a sequence of captured signal frames for extracting hand pose information for a hand;
a processor configured to receive from the sensor a sequence of captured signal frames for extracting hand pose information for a hand, and to execute at least one trained predictor to extract hand pose information from the received signal frames;
wherein the processor is configured to implement the following operations for at least one defined gesture, wherein the gesture is defined as a time sequence comprising hand poses, wherein each of the hand poses is defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand:
computing truth values for the qualitative propositions using the hand pose information extracted from the received signal frames, and
tracking execution of the gesture, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next, wherein upon completion of the sequence, the processor is configured to trigger a function associated with the gesture.