Gesture recognition with hand-object interaction

Inventors: Lailin Chen (Kirkland, WA); Ashwin Kumar Asoka Kumar Shenoi (San Jose, CA); Daniel J. Brewer (San Jose, CA); Eslam A. Mostafa (San Jose, CA); Itay Bar Yosef (Sunnyvale, CA); Julian K. Shutzberg (San Francisco, CA); Leah M. Gum (Sunol, CA); Martin Meloun (San Jose, CA); Minhaeng Lee (Sunnyvale, CA); Victor Belyaev (San Jose, CA)

Assignee: Apple Inc.

G06F3/017 G06V10/26 G06V40/10 G06V40/28

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12710825
App. No.: 18/369,127
Filed: Sep 15, 2023
Granted: Aug 18, 2026
Kind: B2
Examiner: YI, RINNA
Art Unit: 2179
USPC: 715/863

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Aspects of the subject technology provide improved techniques for gesture recognition. Improved techniques may include detecting and/or classifying an interaction between the body part and another object in a scan of the body part, and then controlling recognition of a gesture based on the interaction. In an aspect, recognition parameters may be selected based on the interaction classification that disable recognition of one or more gestures while not disabling recognition of other gestures.

Claims (35)

1 . A method, comprising:

detecting, by a first device and through one or more sensors of the first device, one or more body parts and one or more other objects other than the first device in a scan of a proximate area of a user;

detecting, by the first device, an interaction in the scan between a body part and at least one of the other objects based on an occlusion detected by the one or more sensors indicating an occlusion of the body part by the at least one of the other objects or an occlusion of the at least one of the other objects by the body part;

classifying, by the first device, the interaction to produce an interaction classification, wherein the interaction classification indicates a probability of a physical interaction between the body part and the at least one of the other objects, the probability being based on the detected occlusion; and

controlling, responsive to the interaction classification and by the first device, gesture recognition on the scan by disabling or reducing gesture recognition of one or more gestures for the body part when the probability satisfies a condition, while maintaining or enabling gesture recognition of the one or more gestures for at least one other body part apart from the body part.

2 . The method of claim 1 , wherein the interaction classification indicates that the body part is holding the at least one of the other objects, and the controlling gesture recognition includes foregoing a performance of a gesture recognition on the scan for at least one gesture based on the interaction classification indicating that the body part is holding the at least one of the other objects.

3 . The method of claim 1 , wherein the interaction classifying distinguishes between a holding interaction classification and an obscuring interaction classification where the holding interaction classification indicates the body part is holding the at least one or more other objects while the obscuring interaction classification indicate the at least one or more other objects is partially obscuring the body part in the scan without being held by the body part.

4 . The method of claim 1 , wherein the controlling the gesture recognition disables recognition of at least one gesture when the probability of an interaction is above a threshold, the probability being derived from occlusion-based interaction analysis in the scan.

5 . The method of claim 1 , wherein the controlling gesture recognition includes a bias toward recognition of a first gesture based on the probability of the interaction classification.

6 . The method of claim 1 , wherein the controlling gesture recognition includes selecting recognition parameters based on the interaction classification derived from the scan that disable recognition of one or more disabled gestures, while allowing recognition of other gestures.

7 . The method of claim 6 , wherein the body part is a hand, and the one or more disabled gestures includes a pinch gesture, where the pinch gesture includes a thumb of the hand moving toward a finger of the hand, wherein the pinch gesture is detected from the scan.

8 . The method of claim 6 , wherein the body part is a hand, and the one or more disabled gestures includes a release gesture, where the release gesture includes a thumb of the hand moving away from touching a finger of the hand, wherein the release gesture is detected from the scan.

9 . The method of claim 1 , wherein the body part includes one or more of: a finger, a hand, an arm, a face.

10 . The method of claim 1 , wherein the body part corresponds to a first limb of the user, and wherein the at least one other body part corresponds to a second limb of the user, the second limb on an opposing part of the user than the first limb.

11 . A device, comprising:

one or more sensors;

a processor, and

a memory storing instructions that when executed by the processor cause the device to:

detect through the one or more sensors of the device one or more body parts and one or more other objects other than the device in a scan of a proximate area of a user;

detect an interaction in the scan between a body part and at least one of the other objects based on an occlusion detected by the one or more sensors indicating an occlusion of the body part by the at least one of the other objects or an occlusion of the at least one of the other objects by the body part;

classify the interaction to produce an interaction classification, wherein the interaction classification indicates a probability of a physical interaction between the body part and the at least one of the other objects, the probability being based on the detected occlusion; and

control, responsive to the interaction classification, gesture recognition on the scan by disabling or reducing gesture recognition of one or more gestures for the body part when the probability satisfies a condition, while maintaining or enabling gesture recognition of the one or more gestures for at least one other body part apart from the body part.

12 . The device of claim 11 , wherein the interaction classification indicates that the body part is holding the at least one of the other objects, and the controlling gesture recognition includes foregoing a performance of a gesture recognition on the scan for at least one gesture based on the interaction classification indicating that the body part is holding the at least one of the other objects.

13 . The device of claim 11 , wherein the interaction classifying distinguishes between a holding interaction classification and an obscuring interaction classification where the holding interaction classification indicates the body part is holding the at least one or more other objects while the obscuring interaction classification indicate the at least one or more other objects is partially obscuring the body part in the scan without being held by the body part.

14 . The device of claim 11 , wherein the controlling the gesture recognition disables recognition of at least one gesture when the probability of an interaction is above a threshold.

15 . The device of claim 11 , wherein the controlling gesture recognition includes a bias toward recognition of a first gesture based on the probability of the interaction classification.

16 . A computer readable memory storing instructions that, when executed by a processor, cause the processor to:

detect, by one or more sensors of a first device, one or more body parts and one or more other objects other than the first device in a scan of a proximate area of a user;

17 . The memory of claim 16 , wherein the interaction classification indicates that the body part is holding the at least one of the other objects, and the controlling gesture recognition includes foregoing a performance of a gesture recognition on the scan for at least one gesture based on the interaction classification indicating that the body part is holding the at least one of the other objects.

18 . The memory of claim 16 , wherein the interaction classifying distinguishes between a holding interaction classification and an obscuring interaction classification where the holding interaction classification indicates the body part is holding the at least one or more other objects while the obscuring interaction classification indicate the at least one or more other objects is partially obscuring the body part in the scan without being held by the body part.

19 . The memory of claim 16 , wherein the controlling the gesture recognition disables recognition of at least one gesture when the probability of an interaction is above a threshold.

20 . The memory of claim 16 , wherein the controlling gesture recognition includes a bias toward recognition of a first gesture based on the probability of the interaction classification.