IP Library Granted Patent US 12,430,860
Granted Patent B2
US 12,430,860 · App. 18/664,036 · Granted Sep 30, 2025

Spatial instructions and guides in mixed reality

Inventors: Tushar Arora (Hollywood, FL); Scott Kramarich (Venice, CA)
Assignee: Magic Leap, Inc.
G06T19/006G06F3/011H04W4/024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,430,860
App. No.
18/664,036
Granted
Sep 30, 2025
Kind
B2
Abstract

Exemplary systems and methods for creating spatial contents in a mixed reality environment are disclosed. In an example, a location associated with a first user in a coordinate space is determined. A persistent virtual content is generated. The persistent virtual content is associated with the first user's associated location. The first user's associated location is determined and is associated with the persistent virtual content. A location of a second user at a second time in the coordinate space is determined. The persistent virtual content is presented to the second user via a display at a location in the coordinate space corresponding to the first user's associated location.

Claims (70)

1. A method comprising:

receiving an input from a first user, the input indicative of a movement of the first user;

determining, via a sensor, a location associated with the first user;

generating content corresponding to the input, the content associated with the movement of the first user and further associated with the location associated with the first user;

determining a first location of a second user;

determining, based on the location associated with the first user and based further on the first location of the second user, whether to present the content to the second user;

in accordance with a determination to present the content to the second user:

presenting, via a display, the content to the second user;

determining whether a second location of the second user is within a threshold distance from the first location of the second user;

in accordance with a determination that the second location of the second user is not within the threshold distance, pausing the presenting the content to the second user;

in accordance with a determination that the second location of the second user is within the threshold distance, forgoing pausing the presenting the content to the second user;

while the content is presented to the second user, determining a location of a third user; and

in accordance with a determination that the location of the third user is within a second threshold distance from the first location of the second user, presenting, via a second display, the content to the third user concurrently with the presenting the content to the second user; and

in accordance with a determination not to present the content to the second user, forgoing presenting the content to the second user.

2. The method of claim 1 , wherein the content comprises a video of the movement of the first user.

3. The method of claim 1 , wherein the input from the first user is received during a recording session.

4. The method of claim 1 , wherein:

the sensor is a sensor of a first wearable head device associated with the first user, and

the display is a display of a second wearable head device associated with the second user.

5. The method of claim 1 , wherein the input comprises at least one of a head movement of the first user, a hand movement of the first user, and a handheld controller input.

6. The method of claim 1 , wherein the sensor comprises at least one of a position sensor and a camera.

7. The method of claim 1 , wherein the location associated with the first user is associated with a virtual marking.

8. The method of claim 1 , wherein the determining whether to present the content to the second user comprises determining, via a second sensor, whether a field of view of the second user includes the location associated with the first user.

9. The method of claim 1 , further comprising:

in accordance with the determination to not present the content to the second user:

determining whether a field of view of the second user includes the location associated with the first user; and

in accordance with a determination that the field of view of the second user includes the location associated with the first user, presenting, via the display, a second content.

10. The method of claim 1 , wherein the location associated with the first user comprises one or more of a physical location of the first user and a virtual location of the first user.

11. The method of claim 1 , further comprising:

associating the content with the location of the third user.

12. A system comprising:

a sensor;

a display; and

one or more processors configured to perform a method comprising:

receiving an input from a first user, the input indicative of a movement of the first user;

determining, via the sensor, a location associated with the first user;

generating content corresponding to the input, the content associated with the movement of the first user and further associated with the location associated with the first user;

determining a first location of a second user;

determining, based on the location associated with the first user and based further on the first location of the second user, whether to present the content to the second user;

in accordance with a determination to present the content to the second user:

presenting, via the display, the content to the second user;

determining whether a second location of the second user is within a threshold distance from the first location of the second user;

in accordance with a determination that the second location of the second user is not within the threshold distance, pausing the presenting the content to the second user;

in accordance with a determination that the second location of the second user is within the threshold distance, forgoing pausing the presenting the content to the second user;

while the content is presented to the second user, determining a location of a third user; and

in accordance with a determination that the location of the third user is within a second threshold distance from the first location of the second user, presenting, via a second display, the content to the third user concurrently with the presenting the content to the second user; and

in accordance with a determination not to present the content to the second user, forgoing presenting the content to the second user.

13. The system of claim 12 , wherein:

the sensor is a sensor of a first wearable head device associated with the first user, and

the display is a display of a second wearable head device associated with the second user.

14. The system of claim 12 , wherein the input comprises at least one of a head movement of the first user, a hand movement of the first user, and a handheld controller input.

15. The system of claim 12 , wherein the sensor comprises at least one of a position sensor and a camera.

16. The system of claim 12 , further comprising a second sensor, wherein the determining whether to present the content to the second user comprises determining, via the second sensor, whether a field of view of the second user includes the location associated with the first user.

17. The system of claim 12 , wherein the location associated with the first user comprises one or more of a physical location of the first user and a virtual location of the first user.

18. The system of claim 12 , wherein the method further comprises:

associating the content with the location of the third user.

19. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

receiving an input from a first user, the input indicative of a movement of the first user;

determining, via a sensor, a location associated with the first user;

generating content corresponding to the input, the content associated with the movement of the first user and further associated with the location associated with the first user;

determining a first location of a second user;

determining, based on the location associated with the first user and based further on the first location of the second user, whether to present the content to the second user;

in accordance with a determination to present the content to the second user:

presenting, via a display, the content to the second user;

determining whether a second location of the second user is within a threshold distance from the first location of the second user;

in accordance with a determination that the second location of the second user is not within the threshold distance, pausing the presenting the content to the second user;

in accordance with a determination that the second location of the second user is within the threshold distance, forgoing pausing the presenting the content to the second user;

while the content is presented to the second user, determining a location of a third user; and

in accordance with a determination that the location of the third user is within a second threshold distance from the first location of the second user, presenting, via a second display, the content to the third user concurrently with the presenting the content to the second user; and

in accordance with a determination not to present the content to the second user, forgoing presenting the content to the second user.

Assignments (2)
SECURITY INTEREST Recorded Oct 24, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073255/0581 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2025
From: ARORA, TUSHAR; KRAMARICH, SCOTT
To: MAGIC LEAP, INC.
Reel/Frame 071186/0753 →
Continuity (5)
Continuation 18324824 · May 26, 2023
Continuation 16987144 · Aug 6, 2020
Provisional Application 62894448 · Aug 30, 2019
Provisional Application 62884153 · Aug 7, 2019
Related Publication 20240296639A1 · Sep 5, 2024
References Cited (89)
US 4852988A · Velez · 1989 [cited by applicant]
US 6433760B1 · Vaissie · 2002 [cited by applicant]
US 6491391B1 · Blum et al. · 2002 [cited by applicant]
US 6847336B1 · Lemelson · 2005 [cited by applicant]
US 6943754B2 · Aughey · 2005 [cited by applicant]
US 6977776B2 · Volkenandt et al. · 2005 [cited by applicant]
US 7212873B1 · Townsend · 2007 [cited by examiner]
US 7347551B2 · Fergason et al. · 2008 [cited by applicant]
US 7488294B2 · Torch · 2009 [cited by applicant]
US 8235529B1 · Raffle · 2012 [cited by applicant]
US 8611015B2 · Wheeler · 2013 [cited by applicant]
US 8638498B2 · Bohn et al. · 2014 [cited by applicant]
US 8696113B2 · Lewis · 2014 [cited by applicant]
US 8733927B1 · Lewis · 2014 [cited by applicant]
US 8733928B1 · Lewis · 2014 [cited by applicant]
US 8929589B2 · Publicover et al. · 2015 [cited by applicant]
US 9010929B2 · Lewis · 2015 [cited by applicant]
US 9274338B2 · Robbins et al. · 2016 [cited by applicant]
US 9292973B2 · Bar-zeev et al. · 2016 [cited by applicant]
US 9323325B2 · Perez et al. · 2016 [cited by applicant]
US 9658473B2 · Lewis · 2017 [cited by applicant]
US 9720505B2 · Gribetz et al. · 2017 [cited by applicant]
US 10013053B2 · Cederlund et al. · 2018 [cited by applicant]
US 10025379B2 · Drake et al. · 2018 [cited by applicant]
US 10185147B2 · Lewis · 2019 [cited by applicant]
US 10375009B1 · Fishman · 2019 [cited by applicant]
US 10467809B2 · Khalid · 2019 [cited by applicant]
US 10580218B1 · Viner · 2020 [cited by applicant]
US 10665028B2 · Simari et al. · 2020 [cited by applicant]
US 10937249B2 · Lynen et al. · 2021 [cited by applicant]
US 10937391B2 · Griswold et al. · 2021 [cited by applicant]
US 11704874B2 · Arora et al. · 2023 [cited by applicant]
US 12020391B2 · Arora · 2024 [cited by applicant]
US 20020002585A1 · Matsuda · 2002 [cited by applicant]
US 20030030597A1 · Geist · 2003 [cited by applicant]
US 20060023158A1 · Howell et al. · 2006 [cited by applicant]
US 20080052624A1 · Roberts · 2008 [cited by examiner]
US 20100214111A1 · Schuler et al. · 2010 [cited by applicant]
US 20110106912A1 · Onda et al. · 2011 [cited by applicant]
US 20110211056A1 · Publicover et al. · 2011 [cited by applicant]
US 20110213664A1 · Osterhout · 2011 [cited by applicant]
US 20120021806A1 · Maltz · 2012 [cited by applicant]
US 20130077147A1 · Efimov · 2013 [cited by applicant]
US 20140195918A1 · Friedlander · 2014 [cited by applicant]
US 20150168731A1 · Robbins · 2015 [cited by applicant]
US 20150215351A1 · Barzuza · 2015 [cited by applicant]
US 20150235451A1 · Schowengerdt · 2015 [cited by applicant]
US 20160093108A1 · Mao et al. · 2016 [cited by applicant]
US 20170243403A1 · Daniels · 2017 [cited by applicant]
US 20190098291A1 · Valdivia et al. · 2019 [cited by applicant]
US 20190107990A1 · Spivack et al. · 2019 [cited by applicant]
US 20190130193A1 · Leppänen et al. · 2019 [cited by applicant]
US 20190287306A1 · Wieser et al. · 2019 [cited by applicant]
US 20190304406A1 · Griswold · 2019 [cited by applicant]
US 20200151961A1 · Viner · 2020 [cited by applicant]
US 20200160602A1 · Ghatak · 2020 [cited by applicant]
CA 2316473A1 · 2001 [cited by applicant]
CA 2362895A1 · 2002 [cited by applicant]
CA 2388766A1 · 2003 [cited by applicant]
CN 101904185A · 2010 [cited by applicant]
CN 106716306A · 2017 [cited by applicant]
CN 109154862A · 2019 [cited by applicant]
JP 2001312745A · 2001 [cited by applicant]
JP 2009271750A · 2009 [cited by applicant]
JP 2016533565A · 2016 [cited by applicant]
WO 2014204905A1 · 2014 [cited by applicant]
Chinese Office Action dated Dec. 25, 2024, for CN Patent Application No. 202080070151.8, with English translation, 57 pages. [cited by applicant]
Japanese Notice of Allowance mailed Jan. 10, 2025, for JP Application No. 2023-109116, with English translation, 6 pages. [cited by applicant]
Mingyang, Liao, etc., Dark Souls—Review, “Are you ready to die?” Oct. 19, 2018, located at: https://www.ign.com.cn/hazhgame/23049/review/hei-an-zhi-hun-ceping?ping=2. [cited by applicant]
UnionSquare, “Dark Souls 2 Scholar of Original Sin”, Discover the New World Series: The connection between the Evil Eye Ring and the bloodstains, Mar. 1, 2017, located at: https://www.bilibili.com/video/BV1yx411k7B4/?sp… [cited by applicant]
Azuma, Ronald T. (Aug. 1997). “A Survey of Augmented Reality,” In Presence: Teleoperators and Virtual Environments 6, 4, Hughes Research Laboratories, Malibu, CA, located at: https://web.archive.org/web/20010604100006/h… [cited by applicant]
Bimber, Oliver et al. (2005). “Spatial Augmented Reality: Merging Real and Virtual Worlds,” A. K. Peters, Ltd., Wellesley, MA. [cited by applicant]
European Search Report dated Sep. 1, 2022, for EP Application No. 20849667.9, seven pages. [cited by applicant]
Final Office Action mailed Dec. 29, 2022, for U.S. Appl. No. 16/987,144, filed Aug. 6, 2020, 18 pages. [cited by applicant]
International Preliminary Report and Written Opinion mailed on Feb. 17, 2022, for PCT Application No. PCT/US2020/045496, filed on Aug. 7, 2020, nine pages. [cited by applicant]
International Search Report and Written Opinion mailed on Oct. 29, 2020, for PCT Application No. PCT/US20/45496, filed on Aug. 7, 2020, eighteen pages. [cited by applicant]
Jacob, R. “Eye Tracking in Advanced Interface Design”, Virtual Environments and Advanced Interface Design, Oxford University Press, Inc. (Jun. 1995). [cited by applicant]
Japanese Notice of Allowance mailed Mar. 1, 2024, for JP Application No. 2022-507405, with English translation, six pages. [cited by applicant]
Japanese Office Action dated Oct. 18, 2023, for JP Application No. 2022-507405, with English translation, thirteen pages. [cited by applicant]
Non-Final Office Action mailed Aug. 4, 2022, for U.S. Appl. No. 16/987,144, filed Aug. 6, 2020, seventeen pages. [cited by applicant]
Non-Final Office Action mailed Dec. 20, 2023, for U.S. Appl. No. 18/324,824, filed May 26, 2023, five pages. [cited by applicant]
Notice of Allowance mailed Apr. 15, 2024, for U.S. Appl. No. 18/324,824, filed May 26, 2023, seven pages. [cited by applicant]
Notice of Allowance mailed Apr. 26, 2023, for U.S. Appl. No. 16/987,144, filed Aug. 6, 2020, 8 pages. [cited by applicant]
Rolland, J. et al., “High-resolution inset head-mounted display”, Optical Society of America, vol. 37, No. 19, Applied Optics, (Jul. 1, 1998). [cited by applicant]
Tanriverdi, V. et al. (Apr. 2000). “Interacting With Eye Movements In Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA 02155, USA, Proceedings of the SIGCHI… [cited by applicant]
Yoshida, A. et al., “Design and Applications of a High Resolution Insert Head Mounted Display”, (Jun. 1994). [cited by applicant]
Japanese Office Action mailed Aug. 14, 2024, for JP Application No. 2023-109116, with English translation, 3 pages. [cited by applicant]
European Notice of Allowance dated Mar. 27, 2025, for EP Application No. 20849667.9, five pages. [cited by applicant]
Chinese Office Action dated Jul. 11, 2025, for CN Patent Application No. 202080070151.8, with English translation, 55 pages. [cited by applicant]