IP Library Granted Patent US 11,740,705
Granted Patent B2
US 11,740,705 · App. 17/666,534 · Granted Aug 29, 2023

Method and system for controlling a machine according to a characteristic of a control object

Inventors: Raffi Bedikian (San Francisco, CA); Jonathan Marsden (San Mateo, CA); Keith Mertens (Oakland, CA); David Holz (San Francisco, CA); Maxwell Sills (San Francisco, CA); Matias Perez (San Francisco, CA); Gabriel Hare (Daly City, CA); Ryan Julian (Berkeley, CA)
Assignee: Ultrahaptics IP Two Limited
G06F3/017G06F3/0325G06F3/04815G06V40/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,740,705
App. No.
17/666,534
Granted
Aug 29, 2023
Kind
B2
Abstract

A method and system are provided for controlling a machine using gestures. The method includes sensing a variation of position of a control object using an imaging system, determining, from the variation, one or more primitives describing a characteristic of a control object moving in space, comparing the one or more primitives to one or more gesture templates in a library of gesture templates, selecting, based on a result of the comparing, one or more gesture templates corresponding to the one or more primitives, and providing at least one gesture template of the selected one or more gesture templates as an indication of a command to issue to a machine under control responsive to the variation.

Claims (55)

1. A method of controlling a machine, comprising:

sensing a variation of position of a control object using an imaging system;

determining, from the variation, one or more primitives describing a characteristic of a control object moving in space;

comparing the one or more primitives to one or more gesture templates in a library of gesture templates;

selecting, based on a result of the comparing, one or more gesture templates corresponding to the one or more primitives;

providing, responsive to the variation, at least one gesture template of the selected one or more gesture templates as an indication of a command to issue to a machine under control;

computationally determining a degree of completion of at least one gesture; and

modifying contents of a display in accordance with the determined degree of completion, such that the display gradually fills in an indicator with color or shading indicating how close motion of the control object is to completing the gesture.

2. The method according to claim 1 , further comprising:

identifying a scale associated with the control object, the scale being indicative of an actual distance traversed by the control object; and

computationally determining a ratio between the scale and a displayed movement corresponding to an action to be displayed on a presentation device;

displaying the action on the presentation device based on the ratio; and

adjusting the ratio based on an external parameter.

3. The method according to claim 2 , wherein the external parameter is a ratio of a pixel distance in captured images to a size, in pixels, of a display screen of the presentation device.

4. The method according to claim 1 , wherein the comparing comprises:

disassembling at least a portion of a trajectory of the control object into a set of frequency components; and

searching for a set of frequency components among the one or more gesture templates.

5. The method according to claim 1 , wherein the comparing comprises:

disassembling at least a portion of a trajectory of the control object into a set of time dependent frequency components; and

searching for a set of time dependent frequency components among the one or more gesture templates stored in the library.

6. The method according to claim 5 , wherein the disassembling comprises applying wavelet analysis to the portion of the trajectory as a signal over time to determine the set of time dependent frequency components.

7. The method according to claim 1 , wherein the selecting comprises:

determining a similarity between the one or more primitives and the one or more gesture templates by applying at least one similarity determiner; and

providing the similarity as an indication of a quality of correspondence between the one or more primitives and the one or more gesture templates.

8. The method according to claim 1 , wherein the selecting comprises performing at least one of scaling and shifting to at least one of the one or more primitives and the one or more gesture templates.

9. The method according to claim 1 , wherein the selecting comprises:

disassembling at least a portion of a trajectory of the control object into a set of frequency components;

filtering the set of frequency components to remove motions associated with jitter; and

searching for a gesture template, of the one or more gesture templates, that matches a frequency component of the filtered set of frequency components.

10. The method according to claim 9 , wherein the filtering comprises applying a Frenet-Serret filter.

11. The method according to claim 1 , wherein determining comprises determining a position or motion of the control object relative to a virtual control construct.

12. The method according to claim 1 , further comprising:

identifying two simultaneous gestures based on variations of positions of the control object;

computationally determining a dominant gesture of the two simultaneous gestures; and

presenting an action on a presentation device based on the dominant gesture.

13. The method according to claim 1 , wherein the providing comprises:

detecting a conflict between a gesture template corresponding to a user-defined gesture and a gesture template corresponding to a predetermined gesture; and

applying a resolution determiner to resolve the conflict by selecting one of the user-defined gesture and conflicting predetermined gesture.

14. The method according to claim 13 , wherein the applying comprises selecting the user-defined gesture when the conflict is between the predetermined gesture and the user-defined gesture.

15. A system comprising a memory and one or more processors, the memory being loaded with computer instruction that, when executed on the one or more processors, cause the one or more processors to implement operations comprising:

sensing a variation of position of a control object using an imaging system;

determining, from the variation, one or more primitives describing a characteristic of a control object moving in space;

comparing the one or more primitives to one or more gesture templates in a library of gesture templates;

selecting, based on a result of the comparing, one or more gesture templates corresponding to the one or more primitives;

providing, responsive to the variation, at least one gesture template of the selected one or more gesture templates as an indication of a command to issue to a machine under control;

computationally determining a degree of completion of at least one gesture; and

modifying contents of a display in accordance with the determined degree of completion, such that the display gradually fills in an indicator with color or shading indicating how close motion of the control object is to completing the gesture.

16. A non-transitory computer readable recording medium having computer instructions recorded thereon, the computer instructions, when executed by one or more processor, causing the one or more processor to implement operations comprising:

sensing a variation of position of a control object using an imaging system;

determining, from the variation, one or more primitives describing a characteristic of a control object moving in space;

comparing the one or more primitives to one or more gesture templates in a library of gesture templates;

selecting, based on a result of the comparing, one or more gesture templates corresponding to the one or more primitives;

providing, responsive to the variation, at least one gesture template of the selected one or more gesture templates as an indication of a command to issue to a machine under control;

computationally determining a degree of completion of at least one gesture; and

modifying contents of a display in accordance with the determined degree of completion, such that the display gradually fills in an indicator with color or shading indicating how close motion of the control object is to completing the gesture.

Assignments (6)
SECURITY INTEREST Recorded Apr 6, 2026
From: SIM IP HXR LLC
To: UNITY MASTER LLC SERIES XIX
Reel/Frame 075365/0907 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 19, 2026
From: ULTRAHAPTICS IP TWO LIMITED
To: SIM IP HXR LLC
Reel/Frame 075127/0545 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 16, 2026
From: ULTRAHAPTICS LIMITED; ULTRAHAPTICS IP LIMITED; ULTRAHAPTICS IP TWO LIMITED; ULTRALEAP LIMITED
To: SIM IP HXR LLC
Reel/Frame 074403/0864 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 13, 2022
From: BEDIKIAN, RAFFI; MARSDEN, JONATHAN; MERTENS, KEITH; HOLZ, DAVID; SILLS, MAXWELL; PEREZ, MATIAS; HARE, GABRIEL A.; JULIAN, RYAN
To: LEAP MOTION, INC.
Reel/Frame 062065/0624 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 13, 2022
From: LEAP MOTION, INC.
To: LMI LIQUIDATING CO. LLC
Reel/Frame 062065/0646 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 13, 2022
From: LMI LIQUIDATING CO. LLC
To: ULTRAHAPTICS IP TWO LIMITED
Reel/Frame 062065/0662 →
Cited By (1)
US 12,693,717