IP Library Granted Patent US 11,798,245
Granted Patent B2
US 11,798,245 · App. 18/152,412 · Granted Oct 24, 2023

Waypoint creation in map detection

Inventors: Javier Antonio Busto (Plantation, FL); Jonathan Brodsky (Fort Lauderdale, FL)
Assignee: MAGIC LEAP, INC.
G06T19/006G06F3/011G06F3/016G06T17/05G06T17/20G06V20/20G06T2210/12
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,798,245
App. No.
18/152,412
Granted
Oct 24, 2023
Kind
B2
Abstract

An augmented reality (AR) device can be configured to generate a virtual representation of a user's physical environment. The AR device can capture images of the user's physical environment to generate a mesh map. The AR device can project graphics at designated locations on a virtual bounding box to guide the user to capture images of the user's physical environment. The AR device can provide visual, audible, or haptic guidance to direct the user of the AR device to look toward waypoints to generate the mesh map of the user's environment.

Claims (50)

1. An augmented reality (AR) system comprising:

an AR display configured to present virtual content to a user of the AR system;

an outward facing camera configured to capture one or more images of an environment of the user; and

a hardware processor in communication with the AR display and the outward facing camera, the hardware processor programmed to:

generate a virtual three-dimensional (3D) bounding area;

determine a first waypoint location on a surface of the 3D bounding area;

generate a guidance indicator configured to direct the user to move the outward facing camera to point toward the first waypoint location;

capture one or more images of a physical environment of the user with the outward facing camera pointed toward the first waypoint location;

analyze the one or more images to identify potential objects in the environment;

in response to identifying a potential object in the environment:

generate at least one additional waypoint location associated with the potential object;

generate at least one additional guidance indicator configured to direct the user to move the outward facing camera to point toward the at least one additional waypoint location; and

capture one or more images of the environment with the outward facing camera pointed toward the at least one additional waypoint location; and

update a mesh map based at least partly on the captured images, wherein the mesh map comprises a virtual representation of a physical environment of the user.

2. The AR system of claim 1 , further comprising:

analyzing the images using one or more object recognizers to identifying potential objects in the environment.

3. The AR system of claim 1 , wherein the potential object in the environment comprises furniture.

4. The AR system of claim 1 , wherein the hardware processor is further programmed to:

determine an initial gaze direction of the user, wherein the first waypoint location on the surface of the 3D bounding area is based at least partly on the initial gaze direction.

5. The AR system of claim 4 , wherein the first waypoint location is substantially an intersection of the initial gaze direction with the 3D bounding area.

6. The AR system of claim 1 , wherein the hardware processor is further programmed to:

determine a plurality of waypoint locations, including the first waypoint location, on a surface of the 3D bounding area, wherein the waypoint locations are positioned substantially uniformly across a surface of the 3D bounding area.

7. The AR system of claim 6 , wherein the at least one additional waypoint location is closer to the first waypoint location than to any of the other plurality of waypoint locations.

8. The AR system of claim 6 , wherein the plurality of waypoint locations are positioned at central locations on respective surfaces of the 3D bounding area.

9. The AR system of claim 8 , wherein the surfaces include walls, ceiling, and floor.

10. The AR system of claim 1 , wherein the hardware processor is further programmed to:

determine a plurality of waypoint locations on each of a plurality of surfaces of the 3D bounding area, wherein a quantity of waypoint locations on a first surface is less than a quantity of waypoint locations on a second surface.

11. The AR system of claim 10 , wherein the first surface comprises a ceiling and the second surface comprises a floor.

12. A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:

generate a virtual three-dimensional (3D) bounding area;

determine a first waypoint location on a surface of the 3D bounding area;

generate and display a virtual guidance indicator on a display of a headset including an outward facing camera, wherein the virtual guidance indicator is configured to direct a user to move so that the outward facing camera points toward the first waypoint location;

capture one or more images of a physical environment of the user with the outward facing camera pointed toward the first waypoint location;

analyze the one or more images to identify potential objects in the environment;

in response to identifying a potential object in the environment:

generate at least one additional waypoint location associated with the potential object;

generate at least one additional guidance indicator configured to direct the user to move the outward facing camera to point toward the at least one additional waypoint location; and

capture one or more images of the environment with the outward facing camera pointed toward the at least one additional waypoint location; and

update a mesh map based at least partly on the captured images, wherein the mesh map comprises a virtual representation of a physical environment of the user.

13. The computer readable medium of claim 12 , wherein the software instructions are further configured to cause the computing system to:

analyze the images using one or more object recognizers to identifying potential objects in the environment.

14. The computer readable medium of claim 12 , wherein the potential object in the environment comprises furniture.

15. The computer readable medium of claim 12 , wherein the software instructions are further configured to cause the computing system to:

determine an initial gaze direction of the user, wherein the first waypoint location on the surface of the 3D bounding area is based at least partly on the initial gaze direction.

16. The computer readable medium of claim 15 , wherein the first waypoint location is substantially an intersection of the initial gaze direction with the 3D bounding area.

17. The computer readable medium of claim 12 , wherein the software instructions are further configured to cause the computing system to:

determine a plurality of waypoint locations, including the first waypoint location, on a surface of the 3D bounding area, wherein the waypoint locations are positioned substantially uniformly across a surface of the 3D bounding area.

18. The computer readable medium of claim 17 , wherein the at least one additional waypoint location is closer to the first waypoint location than to any of the other plurality of waypoint locations.

19. The computer readable medium of claim 17 , wherein the plurality of waypoint locations are positioned at central locations on respective surfaces of the 3D bounding area.

20. The computer readable medium of claim 19 , wherein the surfaces include walls, ceiling, and floor.

Assignments (3)
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073031/0206 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2023
From: BUSTO, JAVIER ANTONIO; BRODSKY, JONATHAN
To: MAGIC LEAP, INC.
Reel/Frame 062909/0517 →
SECURITY INTEREST Recorded Feb 7, 2023
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 062681/0065 →