Dynamic, free-space user interactions for machine control
Embodiments of display control based on dynamic user interactions generally include capturing a plurality of temporally sequential images of the user, or a body part or other control object manipulated by the user, and computationally analyzing the images to recognize a gesture performed by the user. In some embodiments, a scale indicative of an actual gesture distance traversed in performance of the gesture is identified, and a movement or action is displayed on the device based, at least in part, on a ratio between the identified scale and the scale of the displayed movement. In some embodiments, a degree of completion of the recognized gesture is determined, and the display contents are modified in accordance therewith. In some embodiments, a dominant gesture is computationally determined from among a plurality of user gestures, and an action displayed on the device is based on the dominant gesture.
1. A method of controlling a machine, comprising:
sensing a variation of a position of a control object of a user moving in a three-dimensional (3D) space in view of an imaging system separated from the user;
determining, from the variation of the position as sensed, one or more primitives describing at least one motion made in space by the control object, the one or more primitives being determined by:
computationally analyzing the variation of the position of the at least one control object to recognize the one or more gesture primitives including at least a portion of a spatial trajectory describing motion made by the control object by identifying a scale and associating the scale with the portion of the spatial trajectory describing the motion made by the control object, the scale being indicative of an actual distance traversed by the control object;
comparing the one or more primitives to one or more gesture templates in a library of gesture templates;
selecting, based upon a result of the comparing, one or more gesture templates corresponding to the one or more primitives; and
providing at least one gesture template of the selected one or more gesture templates as an indication of a command to issue to a machine under control responsive to the variation of the position.
2. A method according to claim 1 , wherein the sensing of the variation of the position comprises:
capturing a plurality of temporally sequential positions of the at least one control object, and wherein the determining, from the variation of the position, of the one or more primitives comprises:
computationally determining a ratio between the scale and a displayed movement corresponding to an action to be displayed on a presentation device; and
providing for displaying the action on the presentation device adjusted based upon the ratio, wherein the one or more primitives further describes a characteristic of the control object.
3. The method according to claim 2 , wherein the scale is identified by comparing the recognized gesture primitive with records storing information describing gestures in a gesture database, wherein the gesture database comprises a set of electronically stored records each relating a gesture to an input parameter, and wherein the records store information describing gestures as sets of vectors.
4. The method according to claim 2 , further comprising:
periodically determining a distance between the control object that performs the gesture and the imaging system;
adjusting the ratio based upon the distance; and
providing, for display, the action to be displayed resized based upon the ratio.
5. The method according to claim 2 , wherein the computationally analyzing of the captured positions of the at least one control object comprises (i) identifying a position and shape of the at least one control object in images and (ii) reconstructing the position and the shape of the at least one control object in 3D space based on correlations determined between the identified shapes and positions of the at least one control object in the images.
6. The method according to claim 1 , wherein the comparing of the one or more primitives to one or more gesture templates of the library of gesture templates comprises:
disassembling at least a portion of a trajectory into a set of frequency components by applying Fourier analysis to the trajectory portion as a signal over time to determine the set of frequency components; and
searching for the set of frequency components among the one or more gesture templates stored in the library.
7. The method according to claim 1 , wherein the comparing of the one or more primitives to one or more gesture templates in the library of gesture templates comprises:
disassembling at least a portion of a trajectory into a set of frequency components;
fitting a set of one or more functions to a set of frequency components representing at least a portion of a trajectory; and
searching for the set of one or more functions among the one or more gesture templates stored in the library, wherein the fitting of the set of the one or more functions comprises:
fitting a Gaussian function to the set of frequency components.
8. The method according to claim 1 , wherein the comparing of the one or more primitives to one or more gesture templates in the library of gesture templates comprises:
disassembling at least a portion of a trajectory into a set of time dependent frequency components by applying a wavelet analysis to the trajectory portion as a signal over time to determine the set of time dependent frequency components; and
searching for the set of time dependent frequency components among the gesture templates stored in the library.
9. The method according to claim 1 , wherein the comparing of the one or more primitives to one or more gesture templates in the library of gesture templates comprises:
distorting at least a portion of a trajectory based at least in part upon frequency of motion components; and
searching for the at least a portion of a trajectory as distorted among the one or more gesture templates stored in the library.
10. The method according to claim 1 , wherein the selecting comprises:
determining a similarity between the one or more primitives and the selected one or more gesture templates by applying at least one similarity determiner selected from a correlation, a convolution, and a dot product;
performing at least one of scaling and shifting to at least one of the one or more primitives and the selected one or more gesture templates; and
providing the similarity as an indication of quality of correspondence between the one or more primitives and the selected one or more gesture templates.
11. The method according to claim 1 , wherein the selecting comprises:
disassembling at least a portion of a trajectory into a set of frequency components;
filtering, by applying a Frenet-Serret filter, the set of frequency components to remove a motion associated with jitter; and
searching the filtered set of frequency components in the one or more gesture templates stored in the library.
12. The method according to claim 1 , wherein the determining, from the variation of the position, of the one or more primitives comprises determining a position or motion of the at least one control object relative to a virtual control construct.
13. The method according to claim 1 , further comprising:
computationally determining a degree of completion of at least one gesture;
modifying contents of a display in accordance with the determined degree of completion, wherein the contents of the display comprise at least one selected from an icon, a bar, a color gradient, and a color or brightness;
comparing the degree of completion to a threshold value; and
indicating a command to be performed based upon a result of the comparing.
14. The method according to claim 13 , further comprising displaying an action responsive to the gesture in accordance with a physics simulation model based upon the degree of completion, wherein the action is displayed based upon a motion model constructed based upon at least one of a simulated physical force, gravity, and a frictional force.
15. The method according to claim 1 , further comprising:
computationally determining, from a plurality of gestures, a dominant gesture; and
presenting, on a presentation device, an action based on the dominant gesture, wherein the dominant gesture is determined by at least filtering the plurality of gestures and selecting from among vectors, the vector having a largest magnitude being determined to be the dominant gesture, wherein each of the gestures is computationally represented as a trajectory by a vector along six Euler degrees of freedom in Euler space.
16. The method according to claim 1 , wherein the providing of the at least one of the selected one or more gesture templates of possible gestures comprises:
filtering one or more gestures based at least in part upon one or more characteristics to determine a set of gestures of interest; and
providing via an API, the set of gestures of interest,
wherein the characteristics comprise at least one of a configuration, shape, and a disposition of the control object making the gesture.
17. The method according to claim 1 , wherein the providing of the at least one of the selected one or more gesture templates of possible gestures comprises:
detecting a conflict between a gesture template corresponding to a user-defined gesture and a gesture template corresponding to a predetermined gesture; and
applying a resolution determiner to resolve the conflict, wherein the applying of the resolution determiner to resolve the conflict comprises at least one selected from:
ignoring the predetermined gesture when the conflict is between the predetermined gesture and the user-defined gesture; and
providing the user-defined gesture when the conflict is between the predetermined gesture and a user-defined gesture.
18. A system enabling dynamic user interactions with a device having a display screen, the system comprising:
at least one sensory device separated from a user and configured to capture positions and motions of a control object of the user moving in a three-dimensional (3D) space;
a gesture database comprising a set of electronically stored records, each of the records relating a gesture to an input parameter; and
one or more processors coupled to the sensory device and a memory storing computer program instructions that, when executed, cause the one or more processors to implement actions including:
capturing a plurality of temporally sequential positions of the at least one control object;
analyzing the positions of the at least one control object to recognize a gesture performed by the user;
comparing the recognized gesture with records in the gesture database to identify an input parameter associated therewith, the input parameter corresponding to an action for display on the display screen in accordance with a determined ratio between (i) a scale indicative of an actual gesture distance traversed by the control object in performance of the gesture and (ii) a displayed movement corresponding to the action; and
providing the action for display adjusted based upon the ratio.
19. A non-transitory computer readable storage medium impressed with computer program instructions to control a machine, the computer program instructions, when executed by a processor, cause the processor to implement actions including:
sensing a variation of a position of a control object of a user moving in a three-dimensional (3D) space in view of an imaging system separated from the user;
determining, from the variation of the position as sensed, one or more primitives describing at least one motion made in space by the control object the one or more primitives being determined by:
computationally analyzing the variation of the position of the at least one control object to recognize the one or more gesture primitives including at least a portion of a spatial trajectory describing motion made by the control object by identifying a scale and associating the scale with the portion of the spatial trajectory describing the motion made by the control object, the scale being indicative of an actual distance traversed by the control object;
comparing the one or more primitives to one or more gesture templates in a library of gesture templates;
selecting, based upon a result of the comparing, one or more gesture templates corresponding to the one or more primitives; and
providing at least one gesture template of the selected one or more gesture templates gestures as an indication of a command to issue to a machine under control responsive to the variation of the position.
20. The method of claim 1 ,
further comprising associating a relevant part of the user's body with the determined one or more primitives,
wherein the selecting of the one or more gesture templates selects the one or more gesture templates in dependence upon the relevant part of the user's body associated with the determined one or more primitives.