IP Library Granted Patent US 12,079,946
Granted Patent B2
US 12,079,946 · App. 18/467,064 · Granted Sep 3, 2024

Waypoint creation in map detection

Inventors: Javier Antonio Busto (Plantation, FL); Jonathan Brodsky (Fort Lauderdale, FL)
Assignee: MAGIC LEAP, INC.
G06T19/006G06F3/011G06F3/016G06T17/05G06T17/20G06V20/20G06T2210/12
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,079,946
App. No.
18/467,064
Granted
Sep 3, 2024
Kind
B2
Abstract

An augmented reality (AR) device can be configured to generate a virtual representation of a user's physical environment. The AR device can capture images of the user's physical environment to generate a mesh map. The AR device can project graphics at designated locations on a virtual bounding box to guide the user to capture images of the user's physical environment. The AR device can provide visual, audible, or haptic guidance to direct the user of the AR device to look toward waypoints to generate the mesh map of the user's environment.

Claims (45)

1. An augmented reality (AR) system comprising:

an AR display configured to present virtual content to a user of the AR system;

an outward facing camera configured to capture one or more images of an environment of the user; and

a hardware processor in communication with the AR display and the outward facing camera, the hardware processor programmed to:

generate a virtual three-dimensional (3D) bounding area;

determine a first waypoint location on a surface of the 3D bounding area;

display a first waypoint at the first waypoint location on the surface of the 3D bounding area;

generate and display a guidance indicator on or near the surface of the 3D bounding area configured to direct the user to move the outward facing camera to point toward the first waypoint location;

capture one or more images of a physical environment of the user when the outward facing camera is pointed toward the first waypoint location; and

update or generate a mesh map based at least partly on the captured images, wherein the mesh map comprises a virtual representation of a physical environment of the user.

2. The AR system of claim 1 , wherein the outward facing camera is generally oriented to correspond to an observation direction of the user, and wherein the guidance indicator is displayed in general alignment with the observation direction.

3. The AR system of claim 2 , wherein the guidance indicator is a virtual reticle.

4. The AR system of claim 2 , wherein the guidance indicator includes a guiding graphic configured to inform the user where the first waypoint location is relative to the guidance indicator.

5. The AR system of claim 4 , wherein the guiding graphic is depicted as an arrow oriented towards the first waypoint location.

6. The AR system of claim 2 , wherein the guidance indicator follows the observation direction of the user.

7. The AR system of claim 6 , wherein the hardware processor is configured to capture the one or more images of the physical environment of the user when the guidance indicator is aligned with the first waypoint in which the outward facing camera is pointed toward the first waypoint location.

8. The AR system of claim 7 , wherein, upon capturing the one or more image of the physical environment when the outward facing camera is pointed towards the first waypoint location, the hardware processor is configured to generate and display a success graphic on or near the surface of the 3D bounding area.

9. The AR system of claim 8 , wherein the hardware processor is configured to replace the display of the first waypoint with the success graphic upon the capturing the one or more image of the physical environment when the outward facing camera is pointed towards the first waypoint location.

10. The AR system of claim 7 , wherein, upon the updating or generating of the mesh map based at least partly on the captured images, the hardware processor is configured to project a graphical representation of the mapped points of the mesh map that were successful mapped.

11. The AR system of claim 7 , wherein, upon a failure to capture the one or more image of the physical environment when the outward facing camera is pointed towards the first waypoint location, the hardware processor is configured to generate and display a failure graphic on or near the surface of the 3D bounding area.

12. The AR system of claim 1 , wherein, upon the capturing of the one or more images of a physical environment of the user when the outward facing camera is pointed toward the first waypoint location, the hardware processor is configured to:

determine a second waypoint location on the surface of the 3D bounding area;

display a second waypoint at the second waypoint location on the surface of the 3D bounding area;

generate and display a further guidance indicator on or near the surface of the 3D bounding area configured to direct the user to move the outward facing camera to point toward the second waypoint location;

capture one or more further images of the physical environment of the user when the outward facing camera is pointed toward the second waypoint location; and

update the mesh map based at least partly on the further captured images.

13. A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by a hardware computer processor to cause a computing system to perform operations comprising:

generate a virtual three-dimensional (3D) bounding area;

determine a first waypoint location on a surface of the 3D bounding area;

display a first waypoint at the first waypoint location on the surface of the 3D bounding area on a display of a headset including an outward facing camera;

generate and display a guidance indicator on or near the surface of the 3D bounding area configured to direct a user to move the outward facing camera to point toward the first waypoint location;

capture one or more images of a physical environment of the user when the outward facing camera is pointed toward the first waypoint location; and

update or generate a mesh map based at least partly on the captured images, wherein the mesh map comprises a virtual representation of a physical environment of the user.

14. The computer readable medium of claim 13 , wherein the outward facing camera is generally oriented to correspond to an observation direction of the user, and wherein the guidance indicator is displayed in general alignment with the observation direction.

15. The computer readable medium of claim 14 , wherein the guidance indicator includes a guiding graphic configured to inform the user where the first waypoint location is relative to the guidance indicator.

16. The computer readable medium of claim 14 , wherein the guidance indicator follows the observation direction of the user.

17. The computer readable medium of claim 16 , wherein the software instructions are configured to cause the computing system to capture the one or more images of the physical environment of the user when the guidance indicator is aligned with the first waypoint in which the outward facing camera is pointed toward the first waypoint location.

18. The computer readable medium of claim 17 , wherein, upon capturing the one or more image of the physical environment when the outward facing camera is pointed towards the first waypoint location, the software instructions are further configured to cause the computing system to: generate and display a success graphic on or near the surface of the 3D bounding area.

19. The computer readable medium of claim 14 , wherein, upon the updating or generating of the mesh map based at least partly on the captured images, the software instructions are further configured to cause the computing system to: project a graphical representation of the mapped points of the mesh map that were successful mapped.

20. The computer readable medium of claim 13 , wherein, upon the capturing of the one or more images of a physical environment of the user when the outward facing camera is pointed toward the first waypoint location, the software instructions are further configured to cause the computing system to:

determine a second waypoint location on the surface of the 3D bounding area;

display a second waypoint at the second waypoint location on the surface of the 3D bounding area;

generate and display a further guidance indicator on or near the surface of the 3D bounding area configured to direct the user to move the outward facing camera to point toward the second waypoint location;

capture one or more further images of a physical environment of the user when the outward facing camera is pointed toward the second waypoint location; and

update the mesh map based at least partly on the further captured images.

Assignments (3)
SECURITY INTEREST Recorded Oct 28, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073387/0487 →
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073031/0206 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 14, 2023
From: BUSTO, JAVIER ANTONIO; BRODSKY, JONATHAN
To: MAGIC LEAP, INC.
Reel/Frame 064909/0554 →