Granted Patent B2

US 12,386,430 · App. 18/759,735 · Granted Aug 12, 2025

Inventors: Alex Marcolina (Los Angeles, CA); David S. Holz (San Francisco, CA)

Assignee: ULTRAHAPTICS IP TWO LIMITED

G06F3/017G06F1/163G06F3/011G06F3/0304G06F3/04815G06T17/10G06T19/006G06T19/20G02B27/017G06T2215/16G06T2219/2016

Abstract

Claims (45)

1. A method of manipulating at least one virtual object using real motions of one or more hands including at least a first hand in a three-dimensional (3D) sensory space, the method including:

obtaining one or more images of at least the first hand in the three-dimensional (3D) sensory space and sensing a location thereof;

incorporating at the location as sensed, one or more images of at least the first hand into a virtual reality scene;

detecting a multiple sub-component free-form gesture made by multiple subcomponents of at least one hand in the 3D sensory space in virtual contact with a virtual object;

determining, in response to the detecting, whether the multiple sub-component free-form gesture is a multiple sub-component virtual contact resulting in grasping or pinching of the virtual object; and

responsive to determining that the multiple sub-component free-form gesture is a multiple sub-component virtual contact resulting in grasping or pinching of the virtual object, linking motion of at least one of the hands to manipulation of the virtual object.

2. The method of claim 1 , further including outlining with an outline, a modeled position of the location of at least the first hand and incorporating the outline into the virtual reality scene.

3. The method of claim 2 , further including changing an appearance of an outline upon detection of a discrepancy between at least one image of at least the first hand and the outline.

4. The method of claim 1 , further including:

in response to detecting the pinch action of the first finger and the thumb of the first hand in the 3D sensory space in virtual contact with a virtual object in a virtual space, depicting visually, a virtual contact between a portion of the first hand and a portion of the virtual object; and

rendering a first virtual pinch force image positioned between the thumb and finger while the pinch action continues.

5. The method of claim 4 , further including:

responsive to detecting a pinch action between a thumb and finger of a second hand and rendering a second virtual pinch force image positioned between the thumb and the finger while the pinch action continues;

sensing a movement of at least one of the first and second hands that increases a separation distance between the first and second hands, while sensing continuing pinching actions of both the first and second hands; and

rendering a new virtual object between the first virtual pinch force image and second virtual pinch force image, responsive to detected increase in the separation distance, wherein at least a size of the new virtual object is responsive to the separation distance.

6. The method of claim 5 , wherein an orientation of the new virtual object is responsive to positions of the first and second hands.

7. The method of claim 1 , wherein

determining whether the multiple sub-component free-form gesture is a multiple sub-component virtual contact and resulting in grasping of the virtual object, the determining further including taking into account at least:

an angular relationship of a normal to a surface of at least one of the hands to a proximate surface of the virtual object;

separations between tips of at least one of the hands and the proximate surface;

a gesture rate at which at least one of the hands closes on the virtual object;

a posture, whether suitable for grasping the virtual object or incompatible with grasping; and

a linear velocity of a surface of at least one of the hands relative to the virtual object; and

responsive to determining that the multiple sub-component free-form gesture is a multiple sub-component virtual contact and resulting in grasping of the virtual object, linking motion of at least one of the hands to manipulation of the virtual object.

8. The method of claim 5 , further including responsive to detecting persistence of the multiple sub-component virtual contact and a decreased proximity between the first hand and the second hand in the 3D sensory space, depicting visually, the multiple sub-component virtual contact and resulting stretching of the virtual object.

9. The method of claim 8 , further including responsive to stretching of the virtual object beyond a predetermined threshold, depicting visually, a modification of the virtual object.

10. The method of claim 5 , further including responsive to stretching of the new virtual object beyond a predetermined threshold, depicting visually, another virtual object.

11. The method of claim 7 , further including determining whether the multiple sub-component free-form gesture is intended to interact with the virtual object based upon a maximum separation between tips of at least one of hand and the proximate surface.

12. The method of claim 7 , further including determining whether the multiple sub-component free-form gesture is intended to interact with the virtual object based upon a rotational velocity about a normal to a surface of at least one hand in the 3D sensory space.

13. The method of claim 1 , further including: detecting a second hand and determining a dominant hand from the hands based on an earliest detection of a three sub-component free-form gesture in the 3D sensory space in virtual contact with a virtual object; and

depicting visually, the virtual contact and resulting grasping of the virtual object by the dominant hand.

14. The method of claim 1 , further including: in response to the detecting, depicting visually, a multiple sub-component virtual contact resulting in a grasping or a pinching of the virtual object by at least one of hand.

15. A system, including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to implement a method of:

obtaining one or more images of at least a first hand in a three-dimensional (3D) sensory space and sensing a location thereof;

incorporating at the location as sensed, one or more images of at least the first hand into a virtual reality scene;

detecting a multiple sub-component free-form gesture made by multiple subcomponents of at least one hand in the 3D sensory space in virtual contact with a virtual object;

determining, in response to the detecting, whether the multiple sub-component free-form gesture is a multiple sub-component virtual contact resulting in grasping or pinching of the virtual object; and

responsive to determining that the multiple sub-component free-form gesture is a multiple sub-component virtual contact resulting in grasping or pinching of the virtual object, linking motion of at least one of the hands to manipulation of the virtual object.

16. A non-transitory computer readable memory having instructions stored thereon, which instructions when executed by a processor implement a method of:

obtaining one or more images of at least a first hand in a three-dimensional (3D) sensory space and sensing a location thereof;

incorporating at the location as sensed, one or more images of at least the first hand into a virtual reality scene;

detecting a multiple sub-component free-form gesture made by multiple subcomponents of at least one hand in the 3D sensory space in virtual contact with a virtual object;

determining, in response to the detecting, whether the multiple sub-component free-form gesture is a multiple sub-component virtual contact resulting in grasping or pinching of the virtual object; and

responsive to determining that the multiple sub-component free-form gesture is a multiple sub-component virtual contact resulting in grasping or pinching of the virtual object, linking motion of at least one of the hands to manipulation of the virtual object.

Assignments (6)

Continuity (7)

References Cited (400)