Granted Patent B2

US 10,600,248 · App. 15/681,251 · Granted Mar 24, 2020

Inventor: David S. Holz (San Francisco, CA)

Assignee: Ultrahaptics IP Two Limited

G06T19/006G02B27/0172G06F3/017G09G5/18G02B2027/0178

Abstract

Claims (46)

1. A system, including:

a head mounted device (HMD) including a presentation mechanism to provide an augmented hybrid experience to a user wearing the head mounted device;

a sensor coupled to the HMD including at least one camera to capture a sequence of images of a physical real environment;

a controller including a processor and memory storing instructions that when executed provide smooth transitioning between an immersive virtual environment and a convergent mixed reality environment during the augmented hybrid experience, wherein the augmented hybrid experience includes selecting between:

(i) an immersive virtual environment including virtual presentation consisting of virtual objects; wherein some virtual objects correspond to sensed physical objects sensed by the sensor; and

(ii) a convergent mixed reality environment including a near real time display of the physical real environment surrounding the user; and

responsive to a first command input received, automatically interrupting the immersive virtual environment being presented to the user and superimposing at least one virtual object of the immersive virtual environment into a generated data stream including near real time physical real environment to generate data representing a mixed reality environment during the augmented hybrid experience; and

responsive to a second command input received, automatically interrupting the mixed reality environment being presented to the user and resuming the immersive virtual environment during the augmented hybrid experience;

wherein the first command input and the second command input are determined using positions of a physical object sensed at time t 0 and time t 1 by the sensor coupled to the HMD and distinguishing motion made by the physical object sensed from motion of the sensor coupled to the HMD and interpreting the motion made by the physical object sensed to indicate a command.

2. The system of claim 1 , wherein a command input is automatically triggered in response to a free-form gesture.

3. The system of claim 1 , wherein a command input is automatically triggered in response to an audio signal.

4. The system of claim 1 , wherein a command input is automatically triggered in response to a vibrational signal.

5. The system of claim 1 , wherein a command input is automatically triggered in response to an optical signal.

6. The system of claim 1 , further including, simultaneously, manipulating in response a command input at least one virtual object of the immersive virtual environment and at least one physical object of the physical real environment during the augmented hybrid experience.

7. A method of smoothly transitioning between an immersive virtual environment and a convergent mixed reality environment during an augmented hybrid experience, the method comprising:

using a sensor having at least one camera coupled to a head mounted device (HMD) to capture a sequence of images of a physical real environment in convergence with presenting to a user wearing the HMD an immersive virtual environment overlaid upon the physical real environment during an augmented hybrid experience; and

responsive to a first command input received, automatically, interrupting the immersive virtual environment being presented to the user and superimposing at least one virtual object of the immersive virtual environment in a generated data stream including near real time physical real environment to generate data representing a mixed reality environment during the augmented hybrid experience; and

responsive to a second command input received, automatically interrupting the mixed reality environment being presented to the user and resuming the immersive virtual environment during the augmented hybrid experience;

wherein the first command input and the second command input are determined using positions of a physical object sensed at time t 0 and time t 1 by the sensor coupled to the HMD and distinguishing motion made by the physical object sensed from motion of the sensor coupled to the HMD and interpreting the motion made by the physical object sensed to indicate a command.

8. The method of claim 7 , wherein the mixed reality environment includes at least one virtual object of the immersive virtual environment and at least one physical object of the physical real environment.

9. The method of claim 7 , wherein a command input is automatically triggered in response to a free-form gesture.

10. The method of claim 7 , wherein a command input is automatically triggered in response to an audio signal.

11. The method of claim 7 , wherein a command input is automatically triggered in response to a vibrational signal.

12. The method of claim 7 , wherein a command input is automatically triggered in response to an optical signal.

13. The method of claim 7 , the method further comprising:

interpreting the motion made by the physical object sensed to indicate a third command; and

automatically triggering a pass through mode of the HMD in response to third command input, wherein the pass through mode interrupts the immersive virtual environment and substitutes a live feed of the physical real environment in the augmented hybrid experience.

14. The method of claim 7 , further including:

at a first time t 0 , using the sensor attached to the HMD, sensing a first position for at least one physical object in a first reference frame of the physical real environment, including tracking portions of the physical object;

causing display of a first virtual representation of the physical object at the first position, wherein the first virtual representation is rendered in the immersive virtual environment of the HMD;

at a second time t 1 , sensing, in the physical real environment, a second position of the physical object and at least some of the portions different from the first position responsive to repositioning of the physical real environment and the sensor due to body movement of a user wearing the HMD, wherein the physical object has not moved in the physical real environment between t 0 and t 1 ; and

causing display of a second virtual representation of the physical object at an actual second position.

15. The method of claim 14 , wherein causing display of a second virtual representation of the physical object at an actual second position further includes:

sensing motion of the sensor and calculating a second reference frame that accounts for repositioning of the sensor;

calculating a transformation that renders the first position in the first reference frame and the second position in the second reference frame into a common reference frame; and

transforming the first position and the second position of the physical object into the common reference frame, wherein the common reference frame has a fixed point of reference and an initial orientation of axes, whereby the second position sensed is transformed to the actual second position.

16. The method of claim 15 , wherein the common reference frame is a world reference frame that does not change as the sensor is repositioned.

17. The method of claim 15 , wherein the common reference frame is the second reference frame.

18. The method of claim 15 , wherein the transforming the first position and second position of the physical object into the common reference frame further includes applying an affine transformation.

19. The method of claim 15 , further including determining an orientation of the physical object at the first position with respect to the first reference frame and causing the display of the physical object accordingly.

20. The method of claim 15 , further including determining an orientation of the physical object at the second position with respect to the second reference frame and causing the display of the physical object accordingly.

21. A non-transitory computer readable medium storing computer instructions for smoothly transitioning between an immersive virtual environment and a convergent mixed reality environment during an augmented hybrid experience, which instructions when executed by one or more processors perform a method comprising:

using a sensor having at least one camera coupled to a head mounted device (HMD) to capture a sequence of images of a physical real environment in convergence with presenting to a user wearing the HMD an immersive virtual environment overlaid upon the physical real environment during an augmented hybrid experience; and

responsive to a first command input received, automatically, interrupting the immersive virtual environment being presented to the user and superimposing at least one virtual object of the immersive virtual environment in a generated data stream including near real time physical real environment to generate data representing a mixed reality environment during the augmented hybrid experience; and

responsive to a second command input received, automatically interrupting the mixed reality environment being presented to the user and resuming the immersive virtual environment during the augmented hybrid experience;

wherein the first command input and the second command input are determined using positions of a physical object sensed at time t 0 and time t 1 by the sensor coupled to the HMD and distinguishing motion made by the physical object sensed from motion of the sensor coupled to the HMD and interpreting the motion made by the physical object sensed to indicate a command.

Assignments (12)

Continuity (3)

Cited By (3)

US 12,299,207 US 12,505,522 US 12,556,674