IP Library Granted Patent US 12,405,674
Granted Patent B2
US 12,405,674 · App. 18/827,206 · Granted Sep 2, 2025

User-defined virtual interaction space and manipulation of virtual cameras with vectors

Inventors: Isaac Cohen (Oakland, CA); Maxwell Sills (San Francisco, CA); Paul Durdik (Foster City, CA)
Assignee: ULTRAHAPTICS IP TWO LIMITED
G06F3/017G06F3/011G06F3/0304G06F3/04815G06F3/04842G06F3/04845G06T15/20G06F2203/04806
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,405,674
App. No.
18/827,206
Granted
Sep 2, 2025
Kind
B2
Abstract

The technology disclosed relates to creating user-defined interaction spaces and modalities in a three dimensional (3D) sensor space in response to control gestures. It also relates to controlling virtual cameras in the 3D sensor space using control gestures and manipulating controls of the virtual cameras through the control gestures. In particular, it relates to defining one or more spatial attributes of the interaction spaces and modalities in response to one or more gesture parameters of the control gesture. It also particularly relates to defining one or more visual parameters of a virtual camera in response to one or more gesture parameters of the control gesture.

Claims (44)

1. A method of manipulating virtual objects in a three-dimensional (3D) sensor space, the method including:

receiving a control gesture that makes swirling motions in a 3D sensor space;

obtaining a virtual vector field using the control gesture received, wherein the virtual vector field is a vortex;

receiving a subsequent control gesture; and

providing for display, the vortex manipulated as a virtual object within the 3D sensor space by interaction of the virtual vector field compounded with the subsequent control gesture.

2. The method of claim 1 , further comprising:

obtaining a size determined for the vortex from the control gesture.

3. The method of claim 2 , wherein one or more virtual objects are (i) brought closer to a center of the vortex, or (ii) pushed away from the center of the vortex, in proportion to interactions obtained from the virtual vector field compounded with object vectors defined on respective virtual objects, based on their respective magnitudes, the method further comprising:

determining based on at least one of (i) the size of the vortex and (ii) a number of virtual objects, a type of gestural effort required to manipulate the vortex as a virtual object.

4. The method of claim 3 , wherein gestural effort is one or more of a flick of a finger, a push of a few fingers, a full handed cup movement.

5. The method of claim 3 , further comprising:

defining the one or more virtual objects in response to one or more subsequent control gestures received.

6. The method of claim 5 , further comprising:

obtaining from the one or more virtual objects, a set of virtual objects determined to be within the vortex based upon the size of the vortex; and

providing for display, members of the set of virtual objects moving in unison responsive to the control gesture.

7. The method of claim 5 , further comprising obtaining an appropriate velocity of motion for the virtual objects determined from a compounding of the size of the vortex and at least one of (i) curling of fingers of a hand and (ii) degrees of freedom between fingers of curled fingers of a hand.

8. The method of claim 1 , further comprising the virtual vector field increasing in strength responsive to continued circling of a control gesture.

9. The method of claim 1 , further comprising obtaining a repulsive vortex in which virtual objects move further apart from one another.

10. A non-transitory computer-readable recording medium having instructions recorded thereon for manipulating virtual objects in a three-dimensional (3D) sensor space, which, when executed on a processor, implement:

receiving a control gesture that makes swirling motions in a 3D sensor space;

obtaining a virtual vector field using the control gesture received, wherein the virtual vector field is a vortex;

receiving a subsequent control gesture; and

providing for display, the vortex manipulated as a virtual object within the 3D sensor space by interaction of the virtual vector field compounded with the subsequent control gesture.

11. The non-transitory computer-readable recording medium of claim 10 , further comprising instructions implementing:

obtaining a size determined for the vortex from the control gesture.

12. The non-transitory computer-readable recording medium of claim 11 , wherein one or more virtual objects are (i) brought closer to a center of the vortex, or (ii) pushed away from the center of the vortex, in proportion to interactions obtained from the virtual vector field compounded with object vectors defined on respective virtual objects, based on their respective magnitudes; and further comprising instructions implementing:

determining based on at least one of (i) the size of the vortex and (ii) a number of virtual objects, a type of gestural effort required to manipulate the vortex as a virtual object.

13. The non-transitory computer-readable recording medium of claim 12 , wherein gestural effort is one or more of a flick of a finger, a push of a few fingers, a full handed cup movement.

14. The non-transitory computer-readable recording medium of claim 11 , further comprising instructions implementing:

defining the one or more virtual objects in response to one or more subsequent control gestures received.

15. The non-transitory computer-readable recording medium of claim 14 , further comprising instructions implementing:

obtaining from the one or more virtual objects, a set of virtual objects determined to be within the vortex based upon the size of the vortex; and

providing for display, members of the set of virtual objects moving in unison responsive to the control gesture.

16. The non-transitory computer-readable recording medium of claim 14 , further comprising instructions implementing: obtaining an appropriate velocity of motion for the virtual objects determined from a compounding of the size of the vortex and at least one of (i) curling of fingers of a hand and (ii) degrees of freedom between fingers of curled fingers of a hand.

17. The non-transitory computer-readable recording medium of claim 10 , further comprising the virtual vector field increasing in strength responsive to continued circling of a control gesture.

18. The non-transitory computer-readable recording medium of claim 10 , further comprising instructions implementing: obtaining a repulsive vortex in which virtual objects move further apart from one another.

19. A smart phone having an interface that manipulates virtual objects in a three-dimensional (3D) sensor space, the smart phone comprising:

a hardware controller storing instructions that, when executed, implement actions including:

receiving a control gesture that makes swirling motions in a 3D sensor space, as viewed by a camera having a particular vantage point;

obtaining a virtual vector field using the control gesture received, wherein the virtual vector field is a vortex;

receiving a subsequent control gesture; and

providing for display, the vortex manipulated as a virtual object within the sensor space by interaction of the virtual vector field compounded with the subsequent control gesture.

20. The smart phone of claim 19 , wherein one or more virtual objects are (i) brought closer to a center of the vortex, or (ii) pushed away from the center of the vortex, in proportion to interactions obtained from the virtual vector field compounded with object vectors defined on respective virtual objects, based on their respective magnitudes; and wherein the actions further include:

determining based on at least one of (i) a size of the vortex and (ii) a number of virtual objects, a type of gestural effort required to manipulate the vortex as a virtual object.

Assignments (6)
SECURITY INTEREST Recorded Apr 6, 2026
From: SIM IP HXR LLC
To: UNITY MASTER LLC SERIES XIX
Reel/Frame 075365/0907 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 19, 2026
From: ULTRAHAPTICS IP TWO LIMITED
To: SIM IP HXR LLC
Reel/Frame 075132/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 16, 2026
From: ULTRAHAPTICS LIMITED; ULTRAHAPTICS IP LIMITED; ULTRAHAPTICS IP TWO LIMITED; ULTRALEAP LIMITED
To: SIM IP HXR LLC
Reel/Frame 074404/0463 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 10, 2025
From: COHEN, ISAAC; SILLS, MAXWELL; DURDIK, PAUL
To: LEAP MOTION, INC.
Reel/Frame 070171/0712 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 10, 2025
From: LEAP MOTION, INC.
To: LMI LIQUIDATING CO. LLC
Reel/Frame 070171/0732 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 10, 2025
From: LMI LIQUIDATING CO. LLC
To: ULTRAHAPTICS IP TWO LIMITED
Reel/Frame 070171/0782 →
Continuity (9)
Continuation 18373243 · Sep 26, 2023
Continuation 17959269 · Oct 3, 2022
Continuation 17378428 · Jul 16, 2021
Continuation 16805639 · Feb 28, 2020
Continuation 16404641 · May 6, 2019
Continuation 15861578 · Jan 3, 2018
Continuation 14572690 · Dec 16, 2014
Provisional Application 61916790 · Dec 16, 2013
Related Publication 20240427426A1 · Dec 26, 2024
References Cited (143)
US 5454043A · Freeman · 1995 [cited by applicant]
US 5554980A · Hashimoto et al. · 1996 [cited by applicant]
US 5594469A · Freeman et al. · 1997 [cited by applicant]
US 5616078A · Oh · 1997 [cited by applicant]
US 6002808A · Freeman · 1999 [cited by applicant]
US 6084979A · Kanade et al. · 2000 [cited by applicant]
US 6160899A · Lee et al. · 2000 [cited by applicant]
US 6191773B1 · Maruno et al. · 2001 [cited by applicant]
US 6222465B1 · Kumar et al. · 2001 [cited by applicant]
US 6788809B1 · Grzeszczuk et al. · 2004 [cited by applicant]
US 6943774B2 · Horiki · 2005 [cited by applicant]
US 7030861B1 · Westerman et al. · 2006 [cited by applicant]
US 7071914B1 · Marks · 2006 [cited by applicant]
US 7095401B2 · Liu et al. · 2006 [cited by applicant]
US 7170492B2 · Bell · 2007 [cited by applicant]
US 7379563B2 · Shamaie · 2008 [cited by applicant]
US 7598942B2 · Underkoffler et al. · 2009 [cited by applicant]
US 7652679B2 · Aratani et al. · 2010 [cited by applicant]
US 7849421B2 · Yoo et al. · 2010 [cited by applicant]
US 7874917B2 · Marks et al. · 2011 [cited by applicant]
US 8144121B2 · Kitaura · 2012 [cited by applicant]
US 8144148B2 · El Dokor et al. · 2012 [cited by applicant]
US 8150102B2 · Zheng et al. · 2012 [cited by applicant]
US 8166421B2 · Magal et al. · 2012 [cited by applicant]
US 8311370B2 · Ha et al. · 2012 [cited by applicant]
US 8325214B2 · Hildreth · 2012 [cited by applicant]
US 8373654B2 · Wang et al. · 2013 [cited by applicant]
US 8379101B2 · Mathe et al. · 2013 [cited by applicant]
US 8457353B2 · Reville et al. · 2013 [cited by applicant]
US 8487871B2 · Langridge et al. · 2013 [cited by applicant]
US 8488246B2 · Border et al. · 2013 [cited by applicant]
US 8502789B2 · Tse et al. · 2013 [cited by applicant]
US 8593402B2 · Stinson, III · 2013 [cited by applicant]
US 8881051B2 · Frey et al. · 2014 [cited by applicant]
US 8933876B2 · Galor et al. · 2015 [cited by applicant]
US 8963867B2 · Narita · 2015 [cited by applicant]
US 9128552B2 · Case et al. · 2015 [cited by applicant]
US 9134800B2 · Maeda · 2015 [cited by applicant]
US 9213890B2 · Huang et al. · 2015 [cited by applicant]
US 9250707B2 · Choi et al. · 2016 [cited by applicant]
US 9256282B2 · Latta et al. · 2016 [cited by applicant]
US 9256288B2 · Lee et al. · 2016 [cited by applicant]
US 9299013B1 · Curlander et al. · 2016 [cited by applicant]
US 9310891B2 · Rafii et al. · 2016 [cited by applicant]
US 9398243B2 · Ryu et al. · 2016 [cited by applicant]
US 9600078B2 · Rafii · 2017 [cited by applicant]
US 9798388B1 · Murali · 2017 [cited by applicant]
US 10275039B2 · Cohen et al. · 2019 [cited by applicant]
US 10901518B2 · Cohen et al. · 2021 [cited by applicant]
US 11500473B2 · Cohen et al. · 2022 [cited by applicant]
US 11567583B2 · Cohen et al. · 2023 [cited by applicant]
US 20020041327A1 · Hildreth et al. · 2002 [cited by applicant]
US 20020064382A1 · Hildreth et al. · 2002 [cited by applicant]
US 20030138130A1 · Cohen et al. · 2003 [cited by applicant]
US 20040046736A1 · Pryor et al. · 2004 [cited by applicant]
US 20040193413A1 · Wilson et al. · 2004 [cited by applicant]
US 20040207597A1 · Marks · 2004 [cited by applicant]
US 20050071864A1 · Denoue et al. · 2005 [cited by applicant]
US 20050285845A1 · Dehlin · 2005 [cited by applicant]
US 20060035710A1 · Festejo et al. · 2006 [cited by applicant]
US 20060239471A1 · Mao et al. · 2006 [cited by applicant]
US 20060252477A1 · Zalewski et al. · 2006 [cited by applicant]
US 20070092118A1 · Tachibana · 2007 [cited by applicant]
US 20070120834A1 · Boillot · 2007 [cited by applicant]
US 20070216894A1 · Garcia et al. · 2007 [cited by applicant]
US 20070220444A1 · Sunday et al. · 2007 [cited by applicant]
US 20080056561A1 · Sawachi · 2008 [cited by applicant]
US 20080141181A1 · Ishigaki et al. · 2008 [cited by applicant]
US 20080244465A1 · Kongqiao et al. · 2008 [cited by applicant]
US 20090031240A1 · Hildreth · 2009 [cited by applicant]
US 20090040215A1 · Afzulpurkar et al. · 2009 [cited by applicant]
US 20090085881A1 · Keam · 2009 [cited by applicant]
US 20090103780A1 · Nishihara et al. · 2009 [cited by applicant]
US 20090217211A1 · Hildreth et al. · 2009 [cited by applicant]
US 20090228841A1 · Hildreth · 2009 [cited by applicant]
US 20090265671A1 · Sachs et al. · 2009 [cited by applicant]
US 20100050134A1 · Clarkson · 2010 [cited by applicant]
US 20100066667A1 · MacDougall et al. · 2010 [cited by applicant]
US 20100083109A1 · Tse et al. · 2010 [cited by applicant]
US 20100230494A1 · Okui · 2010 [cited by applicant]
US 20100234094A1 · Gagner et al. · 2010 [cited by applicant]
US 20100235786A1 · Maizels et al. · 2010 [cited by applicant]
US 20100275159A1 · Matsubara · 2010 [cited by examiner]
US 20100303291A1 · Margolis · 2010 [cited by applicant]
US 20100315336A1 · Butler et al. · 2010 [cited by applicant]
US 20110018795A1 · Jang · 2011 [cited by applicant]
US 20110066984A1 · Li · 2011 [cited by examiner]
US 20110107270A1 · Wang et al. · 2011 [cited by applicant]
US 20110157026A1 · Feng et al. · 2011 [cited by applicant]
US 20110173204A1 · Murillo et al. · 2011 [cited by applicant]
US 20110234840A1 · Klefenz et al. · 2011 [cited by applicant]
US 20110254765A1 · Brand · 2011 [cited by applicant]
US 20110289456A1 · Reville et al. · 2011 [cited by applicant]
US 20120038549A1 · Mandella et al. · 2012 [cited by applicant]
US 20120062558A1 · Lee et al. · 2012 [cited by applicant]
US 20120069168A1 · Huang et al. · 2012 [cited by applicant]
US 20120079420A1 · Arriola · 2012 [cited by applicant]
US 20120086729A1 · Baseley et al. · 2012 [cited by applicant]
US 20120121185A1 · Zavesky · 2012 [cited by applicant]
US 20120179970A1 · Hayes · 2012 [cited by applicant]
US 20120194517A1 · Izadi et al. · 2012 [cited by applicant]
US 20120204133A1 · Guendelman · 2012 [cited by examiner]
US 20120206339A1 · Dahl · 2012 [cited by applicant]
US 20120223882A1 · Galor et al. · 2012 [cited by applicant]
US 20120257035A1 · Larsen · 2012 [cited by applicant]
US 20120268410A1 · King et al. · 2012 [cited by applicant]
US 20120313848A1 · Galor et al. · 2012 [cited by applicant]
US 20120322527A1 · Aoki et al. · 2012 [cited by applicant]
US 20120324332A1 · Zaragoza et al. · 2012 [cited by applicant]
US 20130007616A1 · Bell et al. · 2013 [cited by applicant]
US 20130016896A1 · Seida · 2013 [cited by applicant]
US 20130033483A1 · Im et al. · 2013 [cited by applicant]
US 20130135218A1 · Jain et al. · 2013 [cited by applicant]
US 20130182897A1 · Holz · 2013 [cited by applicant]
US 20130182902A1 · Holz · 2013 [cited by applicant]
US 20130278499A1 · Anderson · 2013 [cited by applicant]
US 20130293454A1 · Jeon et al. · 2013 [cited by applicant]
US 20130293683A1 · Zhou et al. · 2013 [cited by applicant]
US 20130321401A1 · Piemonte et al. · 2013 [cited by applicant]
US 20130342459A1 · Karakotsios et al. · 2013 [cited by applicant]
US 20140089849A1 · Choi et al. · 2014 [cited by applicant]
US 20140137039A1 · Kroeger et al. · 2014 [cited by applicant]
US 20140192206A1 · Holz · 2014 [cited by applicant]
US 20140201666A1 · Bedikian · 2014 [cited by examiner]
US 20140201690A1 · Holz · 2014 [cited by applicant]
US 20140267019A1 · Kuzmin · 2014 [cited by applicant]
US 20140300584A1 · Nishioka · 2014 [cited by applicant]
US 20140317576A1 · Song et al. · 2014 [cited by applicant]
US 20150103004A1 · Cohen et al. · 2015 [cited by applicant]
US 20150135132A1 · Josephson · 2015 [cited by applicant]
US 20150169076A1 · Cohen et al. · 2015 [cited by applicant]
US 20150169175A1 · Cohen et al. · 2015 [cited by applicant]
US 20150169176A1 · Cohen et al. · 2015 [cited by applicant]
US 20150220776A1 · Cronholm · 2015 [cited by applicant]
US 20150277583A1 · Kasahara et al. · 2015 [cited by applicant]
US 20150338998A1 · Chathoth et al. · 2015 [cited by applicant]
US 20150346820A1 · Poupyrev et al. · 2015 [cited by applicant]
US 20150355827A1 · Van Der Westhuizen et al. · 2015 [cited by applicant]
US 20150370416A1 · Nishioka · 2015 [cited by applicant]
US 20170102829A1 · Chikaoka · 2017 [cited by applicant]
Pavlovic, V.I., et al., “Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, No. 7, Jul. 1997, pp. 677-695. [cited by applicant]
Wu et al., Vision-Based Gesture Recognition: A Review, in Gesture-Based Communication in Human-Computer Interaction, Springer, Mar. 1999, 13 pages, pp. 103-115. [cited by applicant]
Bowman, Doug A., et al., “3D User Interfaces: New Directions and Perspectives”, Nov./Dec. 2008, IEEE Computer Society 17pages. [cited by applicant]