Granted Patent B1

US 10,429,923 · App. 15/605,852 · Granted Oct 1, 2019

Inventors: John Adrian Arthur Johnston (San Mateo, CA); Johnathon Scott Selstad (San Francisco, CA); Alex Marcolina (San Francisco, CA)

Assignee: Ultrahaptics IP Two Limited

G06F3/011G02B27/00G06F1/163G06F3/017G06F3/0304G06T19/006G06T2215/16

Abstract

Claims (42)

1. A method of manipulating virtual objects using real motions of one or more hands in a three-dimensional (3D) sensory space, the method including:

receiving in a real time physics engine (RTPE) including a simulation of rigid bodies in a physical system that satisfies a human visual system's expectations for interactions with virtual objects in a virtual environment, a set of virtual object definitions that define a set of virtual objects to the RTPE;

providing to the RTPE a capsule representation of at least one hand determined using a location of the hand sensed from a set of captured images of one or more hands, and selecting ones of the set of virtual objects determined to be within a threshold distance to specific points defined at least on digits of the hand determined from the set of captured images of one or more hands;

determining a one dimensional friction response to a soft contact collision between at least one of the set of virtual objects and a portion of a hand colliding in a single logical frame defined by the RTPE, including:

in a first simulation phase, determining a first solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies and the capsule representation of at least one hand including a one dimensional friction response to a soft contact collision between at least one of the set of virtual objects and a portion of the hand colliding, in opposite direction to a direction of motion being undertaken by the portion of the hand in colliding with the virtual object;

in a second simulation phase, determining a second solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies absent any effects of the hand;

in an integration phase, integrating the first solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies and the capsule representation of at least one hand with the second solution of interactions between the virtual objects in the set of virtual objects simulated as rigid bodies absent effects of the hand such that results of the second solution of interactions are prioritized over results of the first solution of interactions; thereby enabling the set of virtual objects simulated as rigid bodies to act in an integrated solution such that rigid body physical integrity is maintained;

determining a motion to apply to at least one virtual object as a rigid body based upon the integrating the first simulation phase and the second simulation phase; and

presenting across a display of a head mounted device a display of the hand and the virtual object as a rigid body.

2. The method of claim 1 , further implementing the one dimensional friction response with a direction opposite to a velocity of a hand portion colliding with a virtual object encountering a soft contact.

3. The method of claim 1 , further implementing the one dimensional friction response with a magnitude proportional to a velocity of a hand portion colliding with a virtual object encountering a soft contact.

4. The method of claim 1 , further implementing the one dimensional friction response with a magnitude set to a defined selected amount.

5. The method of claim 4 , wherein the defined selected amount is selected to be larger than other forces simulated by the RTPE.

6. The method of claim 1 , further including a first simulation result of the first simulation phase providing expected resultant velocities for virtual objects including at least one expected resultant velocity of at least one virtual object in soft contact with a portion of a hand colliding with the virtual object.

7. The method of claim 1 , wherein a first simulation result of the first simulation phase provide expected resultant velocities for virtual objects including at least one expected resultant velocity of at least one virtual object in soft contact with the portion of a hand colliding with the virtual object, further including:

the second simulation phase discarding results of the first simulation phase whenever attributing the expected resultant velocity to a virtual object causes the virtual object to lose physical integrity.

8. The method of claim 1 , further including capturing the set of captured images of one or more hands in the a three-dimensional (3D) sensory space and sensing a location of at least one hand using a video capturing sensor including at least one camera.

9. The method of claim 1 , further including performing the first simulation phase in a first RTPE and the second simulation phase in a second RTPE, the first RTPE being different from the second RTPE.

10. The method of claim 1 , further including permitting a portion of a hand to partially penetrate a boundary defining a surface of a virtual object during a soft contact.

11. The method of claim 10 , further including: simulating in a brush contact phase a non-soft contact including a frictional force parallel to a surface of a virtual object and between at least one portion of a hand and a surface of the virtual object; and wherein the portion of the hand moves along and approximately parallel to the surface of the virtual object;

detecting a penetration by the portion of the hand into the virtual object exceeding a specified tolerance penetration for the portion of the hand;

responsive to the detecting a penetration exceeding the specified tolerance penetration, switching simulation for the portion of the hand, the virtual object and any other portions of the hand within a specified radius into soft contact collision simulation including the first simulation phase, the second simulation phase and the integration phase.

12. The method of claim 11 , further including starting a timer; and reverting to the brush contact phase when expiry of the timer occurs indicating a state in which no portion of the hand is touching the virtual object.

13. The method of claim 1 , the first simulation phase including receiving positions, velocities and geometry of virtual objects and portions of at least one hand and returning velocities of virtual objects responsive to the hand.

14. A non-transitory computer readable medium impressed with instructions, which instructions, when executed by one or more processors, perform:

receiving in a real time physics engine (RTPE) including a simulation of rigid bodies in a physical system that satisfies a human visual system's expectations for interactions with virtual objects in a virtual environment, a set of virtual object definitions that define a set of virtual objects to the RTPE;

providing to the RTPE a capsule representation of at least one hand determined using a location of the hand sensed from a set of captured images of one or more hands, and selecting ones of the set of virtual objects determined to be within a threshold distance to specific points defined at least on digits of the hand determined from the set of captured images of one or more hands;

determining a one dimensional friction response to a soft contact collision between at least one of the set of virtual objects and a portion of a hand colliding in a single logical frame defined by the RTPE, including:

in a first simulation phase, determining a first solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies and the capsule representation of at least one hand including a one dimensional friction response to a soft contact collision between at least one of the set of virtual objects and a portion of the hand colliding, in opposite direction to a direction of motion being undertaken by the portion of the hand in colliding with the virtual object;

in a second simulation phase, determining a second solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies absent any effects of the hand;

in an integration phase, integrating the first solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies and the capsule representation of at least one hand with the second solution of interactions between the virtual objects in the set of virtual objects simulated as rigid bodies absent effects of the hand such that results of the second solution of interactions are prioritized over results of the first solution of interactions; thereby enabling the set of virtual objects simulated as rigid bodies to act in an integrated solution such that rigid body physical integrity is maintained;

determining a motion to apply to at least one virtual object as a rigid body based upon the integrating the first simulation phase and the second simulation phase; and

presenting across a display of a head mounted device a display of the hand and the virtual object as a rigid body.

15. A system including: an imaging sensor and a controller including one or more processors and a computer readable medium storing instructions, which instructions, when executed by the one or more processors, perform:

receiving in a real time physics engine (RTPE) including a simulation of rigid bodies in a physical system that satisfies a human visual system's expectations for interactions with virtual objects in a virtual environment, a set of virtual object definitions that define a set of virtual objects to the RTPE;

providing to the RTPE a capsule representation of at least one hand determined using a location of the hand sensed from a set of captured images of one or more hands, and selecting ones of the set of virtual objects determined to be within a threshold distance to specific points defined at least on digits of the hand determined from the set of captured images of one or more hands;

determining a one dimensional friction response to a soft contact collision between at least one of the set of virtual objects and a portion of a hand colliding in a single logical frame defined by the RTPE, including:

in a first simulation phase, determining a first solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies and the capsule representation of at least one hand including a one dimensional friction response to a soft contact collision between at least one of the set of virtual objects and a portion of the hand colliding, in opposite direction to a direction of motion being undertaken by the portion of the hand in colliding with the virtual object;

in a second simulation phase, determining a second solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies absent any effects of the hand;

in an integration phase, integrating the first solution of interactions between virtual objects in the set of virtual objects simulated as rigid bodies and the capsule representation of at least one hand with the second solution of interactions between the virtual objects in the set of virtual objects simulated as rigid bodies absent effects of the hand such that results of the second solution of interactions are prioritized over results of the first solution of interactions; thereby enabling the set of virtual objects simulated as rigid bodies to act in an integrated solution such that rigid body physical integrity is maintained;

determining a motion to apply to at least one virtual object as a rigid body based upon the integrating the first simulation phase and the second simulation phase; and

presenting across a display of a head mounted device a display of the hand and the virtual object as a rigid body.

Assignments (12)

Continuity (3)

Cited By (9)

US 12,210,691 US 12,353,632 US 12,357,911 US 12,386,430 US 12,393,398 US 12,449,918 US 12,525,139 US 12,632,120 US 12,684,069