Granted Patent B2

US 10,452,151 · App. 15/917,066 · Granted Oct 22, 2019

Abstract

Claims (99)

1. A machine-implemented method for processing an input gesture, the method comprising:

tracking, using a 3D sensor, user movements including sensing positional information of a portion of a hand in a monitored region of space monitored by the 3D sensor;

using the sensed positional information of the portion of the hand, defining a plurality of distinct user-specific virtual planes, including at least a first user-specific virtual plane defined in space relative to a position of, and corresponding to, a first finger of the hand and a second user-specific virtual plane defined in space relative to a position of, and corresponding to, a second finger of the hand, in the monitored region of space;

detecting, by the 3D sensor, a first finger state of the first finger relative to the corresponding first user-specific virtual plane and a second finger state of the second finger relative to the corresponding second user-specific virtual plane, wherein a finger state for a finger relative to the corresponding user-specific virtual plane defined for the finger is one of: the finger moving closer or further away from the corresponding user-specific virtual plane, and the finger moving on or against the corresponding user-specific virtual plane;

determining an input gesture made by the portion of the hand based on the first finger state and the second finger state;

interpreting the input gesture as a command using the input gesture determined from the first finger state and the second finger state; and

providing the command to a machine for executing an action appropriate to the command.

2. The machine-implemented method of claim 1 , further comprising:

determining the input gesture to be a pinch gesture and interpreting the pinch gesture to be a command indicating a zoom-in command, the determining and interpreting including:

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrating their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

3. The machine-implemented method of claim 1 , further comprising:

determining the input gesture to be a spreading gesture and interpreting spreading gesture to be a command indicating a zoom-out command, the determining and interpreting including:

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger receded from their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

4. The machine-implemented method of claim 1 , further comprising:

determining the input gesture to be a pressure gesture and interpreting the pressure gesture to be a command indicating a command to draw a thicker line on a display, the determining and interpreting including:

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrated their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering content.

5. The method of claim 1 , further comprising:

determining the command to indicate movement of a displayed object on a display responsive to:

determining that at least one of the first finger state and the second finger state indicates that the corresponding finger moved along a surface of the corresponding user-specific virtual plane;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was in a zone specific to entering content.

6. The machine-implemented method of claim 1 , further comprising:

determining from the sensed user movement a speed of the portion of the hand and adjusting gesture-recognition sensitivity based on the speed of the user movement.

7. The machine-implemented method of claim 1 , further comprising:

dynamically personalizing one or more planes by characterizing trajectories of user gestures as vectors in 3-dimensional space, using the vectors to determine a maximal depth extent from gestural movement along an axis extending from the user to a display; and defining a thickness of a plane based upon the maximal depth.

8. The machine-implemented method of claim 7 , further comprising:

revising plane positions as more user movements are detected.

9. The machine-implemented method of claim 1 further comprising:

shifting a plane in space as the user changes position.

10. The machine-implemented method of claim 1 ,

wherein the portion of the hand or other detectable object can be found in more than one zone, the more than one zone comprising a set of zones, and

wherein the method further includes:

determining from the set of zones in which the portion of the hand or other detectable object can be found, a first preferred zone;

assigning the portion of the hand or other detectable object to the first preferred zone; and

reporting the first preferred zone for further processing;

wherein the determining the first preferred zone includes:

applying a set of one or more rules to the set of zones to determine, from the set of zones, the first preferred zone according to the one or more applied rules; and

reporting the first preferred zone for further processing.

11. The machine-implemented method of claim 10 , wherein determining the first preferred zone includes:

applying a zone hierarchy to the set of zones to determine, from the set of zones, a zone highest on the zone hierarchy; and

reporting the zone highest on the zone hierarchy as the first preferred zone for further processing.

12. The machine-implemented method of claim 1 , further comprising determining a correct way to interpret an input detected by the 3D sensor including:

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected;

determining an active program and that the determined zone is a hover zone for the active program; and

interpreting a position or a motion of the portion of the hand or other detectable object based on the active program and the hover zone.

13. The machine-implemented method of claim 1 , further comprising:

interpreting a motion of the portion of the hand or other detectable object as one or more gestures; and

determining from the one or more gestures an input to at least one of (i) an application and (ii) an operating system.

14. A system, including:

an image-capture device including at least one camera;

an image analyzer coupled to the at least one camera, the image analyzer being configured to:

track, using a 3D sensor, user movements including sensing positional information of a portion of a hand in a monitored region of space monitored by the 3D sensor;

using the sensed positional information of the portion of the hand, define a plurality of distinct user-specific virtual planes, including at least a first user-specific virtual plane defined in space relative to a position of, and corresponding to, a first finger of the hand and a second user-specific virtual plane defined in space relative to a position of, and corresponding to, a second finger of the hand, in the monitored region of space;

detect a first finger state of the first finger relative to the corresponding first user-specific virtual plane and a second finger state of the second finger relative to the corresponding second user-specific virtual plane, wherein a finger state for a finger relative to the corresponding user-specific virtual plane defined for the finger is one of: the finger moving closer or further away from the corresponding user-specific virtual plane, and the finger moving on or against the corresponding user-specific virtual plane;

determine an input gesture made by the portion of the hand based on the first finger state and the second finger state;

interpret the input gesture as a command using the input gesture determined from the first finger state and the second finger state; and

provide the command to a machine for executing an action appropriate to the command.

15. The system of claim 14 , wherein the image analyzer is further configured to:

determine the input gesture to be a pinch gesture and interpreting the pinch gesture to be a command indicating a zoom-in command, the determining and interpreting including:

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrating their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

16. The system of claim 14 , wherein the image analyzer is further configured to:

determine the input gesture to be a spreading gesture and interpreting spreading gesture to be a command indicating a zoom-out command, the determining and interpreting including:

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger receded from their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

17. The system of claim 14 , wherein the image analyzer is further configured to:

determine the input gesture to be a pressure gesture and interpreting the pressure gesture to be a command indicating a command to draw a thicker line on a display, the determining and interpreting including:

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrated their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering content.

18. The system of claim 14 , wherein the image analyzer is further configured to:

determine the command to indicate movement of a displayed object on a display responsive to:

determining that at least one of the first finger state and the second finger state indicates that the corresponding finger moved along a surface of the corresponding user-specific virtual plane;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was in a zone specific to entering content.

19. The system of claim 18 , wherein the image analyzer is further configured to:

detect that a finger of the first finger or the second finger penetrated the corresponding user-specific virtual plane preceding detection that the finger state associated with the finger detected motion along the surface of the corresponding user-specific virtual plane, and

determine the command to indicate dragging of the displayed object across the display.

20. A non-transitory computer-readable storage medium storing instructions which when executed by a processor cause the processor to:

track, using a 3D sensor, user movements including sensing positional information of a portion of a hand in a monitored region of space monitored by the 3D sensor;

using the sensed positional information of the portion of the hand, define a plurality of distinct user-specific virtual planes, including at least a first user-specific virtual plane defined in space relative to a position of, and corresponding to, a first finger of the hand and a second user-specific virtual plane defined in space relative to a position of, and corresponding to, a second finger of the hand, in the monitored region of space;

detect, by the 3D sensor, a first finger state of the first finger relative to the corresponding first user-specific virtual plane and a second finger state of the second finger relative to the corresponding second user-specific virtual plane, wherein a finger state for a finger relative to the corresponding user-specific virtual plane defined for the finger is one of: the finger moving closer or further away from the corresponding user-specific virtual plane, and the finger moving on or against the corresponding user-specific virtual plane;

determine an input gesture made by the portion of the hand based on the first finger state and the second finger state;

interpret the input gesture as a command using the input gesture determined from the first finger state and the second finger state; and

provide the command to a machine for executing an action appropriate to the command.

Assignments (13)