IP Library Granted Patent US 12,406,408
Granted Patent B2
US 12,406,408 · App. 18/655,140 · Granted Sep 2, 2025

Augmented reality map curation

Inventors: Griffith Buckley Hazen (San Francisco, CA); Amy DeDonato (Maumee, OH); Ali Shahrokni (San Jose, CA); Ben Weisbih (Rehovot, IL); Vinayram Balakumar (San Francisco, CA)
Assignee: MAGIC LEAP, INC.
G06T11/001G06F3/011G06T3/60G09B29/106G06T2200/24
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,406,408
App. No.
18/655,140
Filed
May 3, 2024
Granted
Sep 2, 2025
Kind
B2
Art Unit
2621
USPC
345/156
Abstract

An augmented reality device may communicate with a map server via an API interface to provide mapping data that may be implemented into a canonical map, and may also receive map data from the map server. A visualization of map quality, including quality indicators for multiple cells of the environment, may be provided to the user as an overlay to the current real-world environment seen through the AR device. These visualizations may include, for example, a map quality minimap and/or a map quality overlay. The visualizations provide guidance to the user that allows more efficient updates to the map, thereby improving map quality and localization of users into the map.

Claims (39)

1. A computing system comprising:

one or more hardware computer processors; and

one or more non-transitory computer readable storage device storing software instructions that, when executed by the one or more hardware computer processors, cause the computing system to:

determine a cell in which a wearable headset is positioned;

determine a cell quality score associated with the cell usable to localize a user into the cell;

determine a cell saturation indicator indicative of whether the user has been positioned within the cell for at least a threshold time period;

determine a cell score indicative of the cell quality score and the cell saturation indicator; and

update a user interface viewable in the wearable headset to indicate the cell score for the cell.

2. The computing system of claim 1 , wherein the cell quality score and the cell saturation indicator are determined by a server remote to the wearable headset and are transmitted to the wearable headset via an application programming interface (API) communication channel.

3. The computing system of claim 2 , wherein the wearable headset initiates the API communication channel by transmitting credentials for directly accessing map data from the server.

4. The computing system of claim 1 , wherein the cell quality score is between 0 and 1, wherein 0 indicates a lowest cell quality and 1 indicates a highest cell quality.

5. The computing system of claim 1 , wherein the software instructions, when executed by the one or more hardware computer processors, cause the computing system to:

in response to determining that the cell saturation indicator is positive, sum the cell quality score and 0.5 to calculate the cell score, with a maximum cell score of 1.

6. The computing system of claim 1 , wherein the software instructions, when executed by the one or more hardware computer processors, cause the computing system to:

in response to determining that the cell saturation indicator is negative; set the cell score equal to the cell quality score.

7. The computing system of claim 1 , wherein the user interface includes a minimap of a portion of a plurality of cells and their corresponding cell scores.

8. The computing system of claim 7 , wherein the minimap is associated with a position of a user input device such that the minimap moves in conjunction with the user input device.

9. The computing system of claim 8 , wherein the minimap is displayed in front of the user input device.

10. The computing system of claim 7 , wherein the software instructions, when executed by the one or more hardware computer processors, cause the computing system to:

in response to movement of the user, rotate the minimap to maintain orientation of the user.

11. The computing system of claim 1 , wherein the user interface includes a map quality overlay with indicators of cell scores overlaying corresponding portions of a real world environment.

12. The computing system of claim 1 , wherein the user interface includes at least one cell quality subscore determined based on images obtained from a particular viewing directions.

13. The computing system of claim 12 , wherein the at least one cell quality subscore includes a plurality of cell quality subscores including a north viewing direction subscore, a south viewing direction subscore, a west viewing direction subscore, and an east viewing direction subscore.

14. The computing system of claim 13 , wherein the cell quality subscores are indicated in the cell.

15. The computing system of claim 14 , wherein the cell quality subscores are indicated as colored areas around an indicator of the cell score.

16. The computing system of claim 7 , wherein the cell scores are indicated in the user interface with colors, wherein lower cell scores are a first color and higher cell scores are a second color.

17. The computing system of claim 16 , wherein the first color is red and the second color is green.

18. The computing system of claim 7 , wherein the user interface indicates the cell with a user icon.

19. The computing system of claim 1 , wherein the cell is one of a plurality of cells in a grid pattern.

20. A computing system comprising:

one or more hardware computer processors; and

one or more non-transitory computer readable storage device storing software instructions that, when executed by the one or more hardware computer processors, cause the computing system to:

display via a wearable headset a minimap indicating quality of canonical map data at each of a plurality of cells of a canonical map, the quality of the canonical map data at each of the plurality of cells characterized by a cell quality score;

determine a cell in which the wearable headset is positioned;

determine a cell saturation indicator indicative of whether the wearable headset has been positioned within the cell for at least a threshold time period;

obtain, via one or more sensors of the wearable headset, images of an environment as a user moves about the environment;

determine updates to the quality of the canonical map data based on the images of the environment;

update a cell score indicative of the cell quality score and the cell saturation indicator; and

update the minimap to indicate the update to the cell score.

Assignments (2)
SECURITY INTEREST Recorded Oct 24, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073255/0581 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2025
From: HAZEN, GRIFFITH BUCKLEY; DEDONATO, AMY; SHAHROKNI, ALI; WEISBIH, BEN; BALAKUMAR, VINAYRAM
To: MAGIC LEAP, INC.
Reel/Frame 070664/0575 →
Continuity (4)
Continuation 18057615 · Nov 21, 2022
Continuation 17153720 · Jan 20, 2021
Provisional Application 62966267 · Jan 27, 2020
Related Publication 20240290009A1 · Aug 29, 2024
References Cited (42)
US 6850221B1 · Tickle · 2005 [cited by applicant]
US 11182614B2 · Sharma · 2021 [cited by applicant]
US 20020129094A1 · Reisman · 2002 [cited by applicant]
US 20060028436A1 · Armstrong · 2006 [cited by applicant]
US 20070081123A1 · Lewis · 2007 [cited by applicant]
US 20090089149A1 · Lerner et al. · 2009 [cited by applicant]
US 20120127062A1 · Bar-Zeev et al. · 2012 [cited by applicant]
US 20120162549A1 · Gao et al. · 2012 [cited by applicant]
US 20130082922A1 · Miller · 2013 [cited by applicant]
US 20130117377A1 · Miller · 2013 [cited by applicant]
US 20130125027A1 · Abovitz · 2013 [cited by applicant]
US 20130208234A1 · Lewis · 2013 [cited by applicant]
US 20130242262A1 · Lewis · 2013 [cited by applicant]
US 20140071539A1 · Gao · 2014 [cited by applicant]
US 20140177023A1 · Gao et al. · 2014 [cited by applicant]
US 20140218468A1 · Gao et al. · 2014 [cited by applicant]
US 20140267420A1 · Schowengerdt et al. · 2014 [cited by applicant]
US 20150016777A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150103306A1 · Kaji et al. · 2015 [cited by applicant]
US 20150178939A1 · Bradski et al. · 2015 [cited by applicant]
US 20150205126A1 · Schowengerdt · 2015 [cited by applicant]
US 20150309263A2 · Abovitz et al. · 2015 [cited by applicant]
US 20150326570A1 · Publicover et al. · 2015 [cited by applicant]
US 20150346495A1 · Welch et al. · 2015 [cited by applicant]
US 20160005229A1 · Lee et al. · 2016 [cited by applicant]
US 20160011419A1 · Gao · 2016 [cited by applicant]
US 20160026253A1 · Bradski et al. · 2016 [cited by applicant]
US 20160246061A1 · Bickerstaff et al. · 2016 [cited by applicant]
US 20170068323A1 · West et al. · 2017 [cited by applicant]
US 20190392643A1 · Busto · 2019 [cited by examiner]
US 20200249749A1 · Akman · 2020 [cited by examiner]
EP 3137976B1 · 2017 [cited by applicant]
JP 2013225918A · 2013 [cited by applicant]
WO WO2019212698A1 · 2019 [cited by applicant]
WO WO2021154558A1 · 2021 [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US21/14138, mailed Apr. 7, 2021. [cited by applicant]
ARToolKit: https://web.archive.org/web/20051013062315/http://www.hitl.washington.edu:80/artoolkit/documentaion/hardware.htm, archived Oct. 13, 2005. [cited by applicant]
Azuma, “A Survey of Augmented Reality,” Teleoperators and Virtual Environments 6, 4 (Aug. 1997), pp. 355-385. https://web.archive.orq/web/20010604100006/http://www.cs.unc.edu/-azuma/ARpresence.pdf. [cited by applicant]
Azuma, “Predictive Tracking for Augmented Realty,” TR95-007, Department of Computer Science, UNG-Chapel Hill, NC, Feb. 1995, 262 pages. [cited by applicant]
Bimber, et al., “Spatial Augmented Reality—Merging Real and Virtual Worlds,” 2005 https://web.media.mit.edu/-raskar/book/BimberRaskarAugmentedRealityBook.pdf. [cited by applicant]
Jacob, “Eye Tracking in Advanced Interface Design,” Human-Computer Interaction Lab Naval Research Laboratory, Washington, D.C. /paper/ in Virtual Environments and Advanced Interface DesiQn, ed. by W. Barfield and T.A. F… [cited by applicant]
Tanriverdi and Jacob, “Interacting With Eye Movements in Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA—paper/Proc. ACM CHI 2000 Human Factors in Computin… [cited by applicant]