Granted Patent B2

US 12,249,035 · App. 18/046,739 · Granted Mar 11, 2025

Inventors: Daniel LeWinn Lehrich (Miami, FL); David C. Lundmark (Los Altos, CA)

Assignee: Magic Leap, Inc.

G06T19/006G06F3/013G06F3/014G06F18/23G06V20/20

Abstract

Claims (51)

1. An augmented reality viewer comprising:

at least one sensor positioned to sense a location of at least one of a plurality of real-world objects;

a storage device;

a set of data on the storage device including a virtual object;

a processor connected to the storage device;

a real-world surface extraction routine on the storage device and executable with the processor to identify first and second portions of a real-world surface among the real-world objects;

a display module on the storage device and executable by the processor to determine a desired display of the virtual object relative to the location of at least one of the real-world objects;

a data stream generator on the storage device and executable by the processor to generate a data stream based on the data and the desired display;

a light generator connected to the processor to receive the data stream and generate light based on the data stream;

a display device positioned to receive the light that is generated and to display the light to a user, wherein the light creates a rendering of the virtual object visible to the user and rendered in accordance with the desired display, wherein the display device is a see-through display device that allows light from the real-world surface to reach an eye of the user; and

a depth creation module on the storage device and executable with the processor to display, according to the desired display, in three-dimensional space, the virtual object to the user on a side of the real-world surface opposing the user and with the first portion of the real-world surface that is identified by the real-world surface extraction routine between the user and the virtual object and the virtual object being displayed to the user instead of the first portion of the real-world surface, while the second portion of the real-world surface that is identified by the real-world surface extraction routine is viewable by the user through the display device, thereby rendering a hole with a rendered periphery in the real-world surface through which the virtual object is rendered for viewing in three-dimensional space.

2. The augmented reality viewer of claim 1 , wherein the real-world re surface is a two-dimensional real-world surface of a wall or a ceiling.

3. The augmented reality viewer of claim 1 , wherein the depth creation module is executable with the processor to display a porthole in the real-world surface to the user through which the virtual object is visible to the user while a portion of the real-world surface is viewable by the user around the porthole.

4. The augmented reality viewer of claim 1 , wherein the virtual object is a three-dimensional virtual object.

5. The augmented reality viewer of claim 1 , further comprising:

a vista placement routine on the storage device and executable with the processor to:

capture a space that includes the real-world objects;

represent the space as a real-world mesh;

collect vertical and horizontal planes from the real-world mesh;

filter the planes by height from a floor, dimensions, orientation, and location relative to the real-world mesh;

spawn a blueprint which includes a portal frame and all the content in the vista at the selection location; and

cut a hole in an occlusion material of the real-world mesh material so that the user can see through the portal into the vista.

6. The augmented reality viewer of claim 1 , further comprising:

a head-worn structure shaped to be worn on a head of the user, wherein the display device is mounted to the head-worn structure and the at least one sensor is of a kind that is suitable to sense movement of the display device due to movement of a head of the user; and

a position adjustment module, executable by the processor, to adjust a location of the virtual object so that, within a view of the user, the virtual object remains stationary relative to the at least one real-world object.

7. The augmented reality viewer of claim 1 , wherein the virtual object is a nomadic life object, further comprising:

a nomadic subroutine on the storage device and executable by the processor to move the nomadic life object relative to the real-world objects.

8. An augmented reality visualization method comprising:

sensing, with at least one sensor, a location of at least one of a plurality of real-world objects;

storing, on a storage device, data including a virtual object;

identifying, with the processor, first and second portions of a real-world surface among the real-world objects;

determining, with a processor, a desired display of the virtual object relative to the location of at least one of the real-world objects;

generating, by the processor, a data stream based on the data and the desired display;

generating, with a light generator, light based on the data stream;

displaying, with a display device, the light that is generated to a user, wherein the light creates a rendering of the virtual object visible to the user and rendered in accordance with the desired display, wherein the display device is a see-through display device that allows light from the real-world surface to reach an eye of the user; and

wherein the processor, according to the desired display, displays, in three-dimensional space, the virtual object to the user on a side of the real-world surface opposing the user and with the first portion of the real-world surface that is identified between the user and the virtual object and the virtual object being displayed to the user instead of the first portion of the real-world surface, while the second portion of the real-world surface that is identified is viewable by the user through the display device, thereby rendering a hole with a rendered periphery in the real-world surface through which the virtual object is rendered for viewing in three-dimensional space.

9. The method of claim 8 , wherein the real-world surface is a two-dimensional real-world surface of a wall or a ceiling.

10. The method of claim 8 , wherein the processor displays a porthole in the real-world surface to the user through which the virtual object is visible to the user while a portion of the real-world surface is viewable by the user around the porthole.

11. The method of claim 8 , wherein the virtual object is a three-dimensional virtual object.

12. The method of claim 8 , further comprising:

capturing, with the processor, a space that includes the real-world objects;

representing, with the processor, the space as a real-world mesh;

collecting, with the processor, vertical and horizontal planes from the real-world mesh;

filtering, with the processor, the planes by height from a floor, dimensions, orientation, and location relative to the real-world mesh;

spawning, with the processor, a blueprint which includes a portal frame and all the content in the vista at the selection location; and

cutting, with the processor, a hole in an occlusion material of the real-world mesh material so that the user can see through the portal into the vista.

13. The method of claim 8 , further comprising:

sensing, with the at least one sensor, movement of the display device due to movement of a head of the user; and

adjusting a location of the virtual object so that, within a view of the user, the virtual object remains stationary relative to the at least one real-world object.

14. The method of claim 8 , wherein the virtual object is a nomadic life object, further comprising:

moving, with the processor, the nomadic life object relative to the real-world objects.

Assignments (4)

Continuity (5)

References Cited (400)