IP Library Granted Patent US 11,514,673
Granted Patent B2
US 11,514,673 · App. 16/938,782 · Granted Nov 29, 2022

Systems and methods for augmented reality

Inventors: Daniel Lewinn Lehrich (Miami, FL); David C. Lundmark (Los Altos, CA)
Assignee: Magic Leap, Inc.
G06V20/20G06F3/013G06F3/014G06K9/6218
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,514,673
App. No.
16/938,782
Granted
Nov 29, 2022
Kind
B2
Abstract

An augmented reality viewer includes components, assemblies, and executable logic to provide a user with the perception of rich augmented reality experiences, including aspects of the aquatic world.

Claims (78)

1. An augmented reality viewer comprising:

at least one sensor positioned to sense a location of at least one of a plurality of real-world objects and a wave movement initiated by a user in a target zone;

a storage device;

a set of data on the storage device including a nomadic life object;

a processor connected to the storage device;

a display module on the storage device and executable by the processor to determine a desired display of the nomadic life object relative to the location of at least one of the real-world objects;

a data stream generator on the storage device and executable by the processor to generate a data stream based on the data and the desired display;

a light generator connected to the processor to receive the data stream and generate light based on the data stream;

a display device positioned to receive the light that is generated and to display the light to the user, wherein the light creates a rendering of the nomadic life object visible to the user and rendered in accordance with the desired display;

a nomadic subroutine on the storage device and executable by the processor to move the nomadic life object relative to the real-world objects;

a wave movement routine on the storage device and executable by the processor to move the nomadic life object out of and away from the target zone in response to and because of the wave movement being sensed by the at least one sensor in the target zone;

a movement module on the storage device and executable with the processor to:

sense, with the at least one sensor, a slow hand movement of a hand of the user in the target zone;

articulate the body of the nomadic life object in the first back-and-forth manner at a low speed out of the target zone in response to the slow speed of the hand movement;

sense, with the at least one sensor, a fast hand movement of the hand of the user;

articulate the body of the nomadic life object in the first back-and-forth manner at a high speed out of the target zone in response to the fast speed of the hand movement; and

move the nomadic life object to stay close to the hand when the hand moves at the slow speed and due to the slow speed, and move the nomadic life object to flee the hand when the hand moves at the fast speed and due to the fast speed.

2. The augmented reality viewer of claim 1 , wherein the data includes a plurality of nomadic life objects, wherein the nomadic subroutine is executable by the processor to move the plurality of nomadic life objects relative to the real-world objects and relative to one another and the wave movement routine on the storage device is executable by the processor to move the plurality of nomadic life object out of and away from the target zone in response to and because of the wave movement being sensed by the at least one sensor in the target zone.

3. The augmented reality viewer of claim 1 , wherein the at least one sensor sensing the wave movement is a camera that detects an image of a hand of the user in the target zone.

4. The augmented reality viewer of claim 1 , further comprising:

a handheld controller, wherein the sensor is mounted to a handheld controller that is held in a hand of the user to sense the hand of the user in the target zone.

5. The augmented reality viewer of claim 1 , further comprising:

an artificial intelligence system on the storage device and executable with the processor to:

sense a wave movement;

associate, in response to sensing the wave movement, the movement of the nomadic life object with the wave movement as an artificial intelligence cluster,

determine a parameter that exists at a first time when the action is sensed,

associate, in response to the determining of the parameter at the first time, the parameter that exists at the first time with the artificial intelligence cluster;

sense a parameter at a second time;

determine, in response to the determining of the parameter at the second time, whether the parameter at the second time is the same as the parameter at the first time; and

if the determination is made that the parameter at the second time is the same as the parameter at the first time then executing the wave movement routine in response to the determination that is made that the parameter at the second time is the same as the parameter at the first time.

6. The augmented reality viewer of claim 5 , wherein the at least one sensor includes an eye tracking camera, wherein the first and second parameters are each a gaze direction as determined by eye tracking of the user using the eye tracking camera.

7. The augmented reality viewer of claim 1 , wherein there are a plurality of nomadic life objects and the nomadic subroutine is executable by the processor to move the plurality of nomadic life object relative to the real-world objects, further comprising:

a personal assistant module on the storage device and executable with the processor to:

select a personal assistant nomadic life object among the plurality of nomadic life objects; and

move at least one of the nomadic life objects other than the personal assistant nomadic life objects with the personal assistant nomadic life object in response to an instruction by the user.

8. The augmented reality viewer of claim 1 , wherein the display device displays the light that is generated to the user while the user views at least one of the real-world objects.

9. The augmented reality viewer of claim 8 , wherein the display device is a see-through display device that allows light from the at least one real-world objects to reach an eye of the user.

10. The augmented reality viewer of claim 1 , further comprising:

a head-worn structure shaped to be worn on a head of the user, wherein the display device is mounted to the head-worn structure and the at least one sensor is of a kind that is suitable to sense movement of the display device due to movement of a head of the user; and

a position adjustment module, executable by the processor, to adjust a location of the nomadic life object so that, within a view of the user, the nomadic life object remains stationary relative to the at least one real-world object.

11. An augmented reality visualization method comprising:

sensing, with at least one sensor, a location of at least one of a plurality of real-world objects;

storing, on a storage device, data including a nomadic life object;

determining, with a processor, a desired display of the nomadic life object relative to the location of at least one of the real-world objects;

generating, by the processor, a data stream based on the data and the desired display;

generating, with a light generator, light based on the data stream;

displaying, with a display device, the light that is generated to a user, wherein the light creates a rendering of the nomadic life object visible to the user and rendered in accordance with the desired display;

moving, with the processor, the nomadic life object relative to the real-world objects;

sensing, with the at least one sensor, a wave movement initiated by the user in a target zone;

moving, with the processor, the nomadic life object out of and away from the target zone in response to and because of the wave movement being sensed by the at least one sensor in the target zone;

sensing, with the at least one sensor, a slow hand movement of a hand of the user in the target zone;

articulating, with the processor, the body of nomadic life object in the first back-and-forth manner at a low speed out of the target zone in response to the slow speed of the hand movement;

sensing, with the at least one sensor, a fast hand movement of the hand of the user;

articulating, with the processor, the body of nomadic life object in the first back-and-forth manner at a high speed out of the target zone in response to the fast speed of the hand movement;

moving, with the processor, the nomadic life object to stay close to the hand when the hand moves at the slow speed and due to the slow speed; and

moving, with the processor, the nomadic life object to flee the hand when the hand moves at the fast speed and due to the fast speed.

12. The method of claim 11 , wherein the data includes a plurality of nomadic life objects, further comprising:

moving, with the processor, the plurality of nomadic life objects relative to the real-world objects and relative to one another and to move the plurality of nomadic life object out of and away from the target zone in response to and because of the wave movement being sensed by the at least one sensor in the target zone.

13. The method of claim 11 , wherein the at least one sensor sensing the wave movement is a camera that detects an image of a hand of the user in the target zone.

14. The method of claim 11 , wherein the sensor is mounted to a handheld controller that is held in a hand of the user to sense the hand of the user in the target zone.

15. The method of claim 11 , further comprising:

sensing, with the at least one sensor, the wave movement;

associating, with the processor, in response to sensing the wave movement, the movement of the nomadic life object with the wave movement as an artificial intelligence cluster;

determining, with the at least one processor, a parameter that exists at a first time when the action is sensed;

associating, with the processor, in response to the determining of the parameter at the first time, the parameter that exists at the first time with the artificial intelligence cluster;

sensing, with the at least one sensor, in response to the determining of the parameter at the second time, a parameter at a second time;

determining, with the processor, whether the parameter at the second time is the same as the parameter at the first time; and

if the determination is made that the parameter at the second time is the same as the parameter at the first time then executing the movement of the nomadic life object in response to the determination that is made that the parameter at the second time is the same as the parameter at the first time.

16. The method of claim 15 , wherein the action is a wave motion initiated by the user and the first and second parameters are each a gaze direction of the user.

17. The method of claim 11 , further comprising:

moving, with the processor, a plurality of nomadic life object relative to the real-world objects;

selecting, with the processor, a personal assistant nomadic life object among the plurality of nomadic life objects; and

moving, with the processor, at least one of the nomadic life objects other than the personal assistant nomadic life objects with the personal assistant nomadic life object in response to an instruction by the user.

18. The method of claim 11 , wherein the display device displays the light that is generated to the user while the views at least one of the real-world objects.

19. The method of claim 18 , wherein the display device is a see-through display device that allows light from the at least one real-world objects to reach an eye of the user.

20. The method of claim 11 , further comprising:

sensing, with the at least one sensor, movement of the display device due to movement of a head of the user; and

adjusting a location of the nomadic life object so that, within a view of the user, the nomadic life object remains stationary relative to the at least one real-world object.

Assignments (5)
SECURITY INTEREST Recorded Oct 28, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073388/0027 →
SECURITY INTEREST Recorded Feb 7, 2023
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 062681/0065 →
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 21, 2022
From: LEHRICH, DANIEL LEWINN
To: MAGIC LEAP, INC.
Reel/Frame 058728/0243 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 19, 2020
From: LEHRICH, DANIEL LEWINN; LUNDMARK, DAVID C.
To: MAGIC LEAP, INC.
Reel/Frame 053543/0066 →
Continuity (4)
Provisional Application 62899678 · Sep 12, 2019
Provisional Application 62881355 · Jul 31, 2019
Provisional Application 62879408 · Jul 26, 2019
Related Publication 20210097286A1 · Apr 1, 2021
Cited By (3)
US 12,211,076 US 12,373,870 US 12,700,118