IP Library › Granted Patent US 12,666,145
Granted Patent B2
US 12,666,145 · App. 18/957,796 · Granted Jun 23, 2026

Depth sensing techniques for virtual, augmented, and mixed reality systems

Inventors: Brian Keith Smith (Wellington, FL); Koon Keong Shee (Miramar, FL); Gregory Michael Link (Charlotte, NC)
Assignee: Magic Leap, Inc.
H04N23/667G01S17/894G06F3/00H04N13/139H04N13/296H04N23/959
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,666,145
App. No.
18/957,796
Filed
Nov 24, 2024
Granted
Jun 23, 2026
Kind
B2
Art Unit
2482
USPC
348/46
Abstract

Techniques for operating a depth sensor are discussed. A first sequence of operation steps and a second sequence of operation steps can be stored in memory on the depth sensor to define, respectively, a first depth sensing mode of operation and a second depth sensing mode of operation. In response to a first request for depth measurement(s) according to the first depth sensing mode of operation, the depth sensor can operate in the first mode of operation by executing the first sequence of operation steps. In response to a second request for depth measurement(s) according to the second depth sensing mode of operation, and without performing an additional configuration operation, the depth sensor can operate in the second mode of operation by executing the second sequence of operation steps.

Claims (62)

1 . A method for operating a depth sensor, the method comprising:

receiving, by the depth sensor, a first request for one or more first depth measurements according to a first depth sensing mode of operation of the depth sensor, wherein the first depth sensing mode of operation is defined by a first sequence of operation steps stored in memory on the depth sensor, the first sequence of operation steps including one or more common operation steps;

in response to the first request, performing the first mode of operation by executing the first sequence of operation steps to generate the one or more first depth measurements;

receiving, by the depth sensor, a second request for one or more second depth measurements according to a second depth sensing mode of operation of the depth sensor, wherein the second depth sensing mode of operation is defined by a combination of the one or more common operations steps and a second sequence of operation steps stored in the memory on the depth sensor; and

in response to the second request and without storing additional operation steps in the memory, performing the second mode of operation by executing the second sequence of operation steps and the one or more common operation steps to generate the one or more second depth measurements,

wherein the first depth sensing mode of operation or the second depth sensing mode of operation is a short-range and high frame rate mode of operation.

2 . The method of claim 1 , wherein the depth sensor is a time-of-flight camera.

3 . The method of claim 1 , wherein the first depth sensing mode of operation corresponds to a first distance range of measurements and the second depth sensing mode of operation corresponds to a second distance range of measurements which is different from the first distance range of measurements.

4 . The method of claim 1 , wherein the one or more first depth measurements and the one or more second depth measurements are usable in a virtual, augmented, or mixed reality display system.

5 . The method of claim 1 , wherein the first depth sensing mode of operation corresponds to a first frame rate and the second depth sensing mode of operation corresponds to a second frame rate which is different than the first frame rate.

6 . The method of claim 1 , wherein the first depth sensing mode of operation corresponds to a first frame rate and the second depth sensing mode of operation corresponds to a second frame rate which is slower than the first frame rate.

7 . The method of claim 1 , wherein at least one of the second sequence of operation steps comprises a delay.

8 . The method of claim 1 , further comprising:

receiving at least one configuration request; and

in response to the at least one configuration request, storing, in the memory, the first sequence of operation steps and the second sequence of operations steps.

9 . The method of claim 1 , wherein the short-range and high frame rate mode of operation is used for sensing depths at ranges less than about 2 m with frame rates greater than about 20 Hz.

10 . A method for operating a depth sensor, the method comprising:

receiving, by the depth sensor, a first request for one or more first depth measurements according to a first depth sensing mode of operation of the depth sensor, wherein the first depth sensing mode of operation is defined by a first sequence of operation steps stored in memory on the depth sensor, the first sequence of operation steps including one or more common operation steps;

in response to the first request, performing the first mode of operation by executing the first sequence of operation steps to generate the one or more first depth measurements;

receiving, by the depth sensor, a second request for one or more second depth measurements according to a second depth sensing mode of operation of the depth sensor, wherein the second depth sensing mode of operation is defined by a combination of the one or more common operations steps and a second sequence of operation steps stored in the memory on the depth sensor; and

in response to the second request and without storing additional operation steps in the memory, performing the second mode of operation by executing the second sequence of operation steps and the one or more common operation steps to generate the one or more second depth measurements,

wherein the first depth sensing mode of operation or the second depth sensing mode of operation is a short-range and low frame rate mode of operation.

11 . The method of claim 10 , wherein the short-range and low frame rate mode of operation is used for sensing depths at ranges less than about 2 m with frame rates less than about 20 Hz.

12 . The method of claim 10 , wherein the depth sensor is a time-of-flight camera.

13 . A method for operating a depth sensor, the method comprising:

receiving, by the depth sensor, a first request for one or more first depth measurements according to a first depth sensing mode of operation of the depth sensor, wherein the first depth sensing mode of operation is defined by a first sequence of operation steps stored in memory on the depth sensor, the first sequence of operation steps including one or more common operation steps;

in response to the first request, performing the first mode of operation by executing the first sequence of operation steps to generate the one or more first depth measurements;

receiving, by the depth sensor, a second request for one or more second depth measurements according to a second depth sensing mode of operation of the depth sensor, wherein the second depth sensing mode of operation is defined by a combination of the one or more common operations steps and a second sequence of operation steps stored in the memory on the depth sensor; and

in response to the second request and without storing additional operation steps in the memory, performing the second mode of operation by executing the second sequence of operation steps and the one or more common operation steps to generate the one or more second depth measurements,

wherein the first depth sensing mode of operation or the second depth sensing mode of operation is a long-range and high frame rate mode of operation.

14 . The method of claim 13 , wherein the long-range and high frame rate mode of operation is used for sensing depths at ranges from about 2-4 m with frame rates greater than about 20 Hz.

15 . The method of claim 13 , wherein the depth sensor is a time-of-flight camera.

16 . A method for operating a depth sensor, the method comprising:

receiving, by the depth sensor, a first request for one or more first depth measurements according to a first depth sensing mode of operation of the depth sensor, wherein the first depth sensing mode of operation is defined by a first sequence of operation steps stored in memory on the depth sensor, the first sequence of operation steps including one or more common operation steps;

in response to the first request, performing the first mode of operation by executing the first sequence of operation steps to generate the one or more first depth measurements;

receiving, by the depth sensor, a second request for one or more second depth measurements according to a second depth sensing mode of operation of the depth sensor, wherein the second depth sensing mode of operation is defined by a combination of the one or more common operations steps and a second sequence of operation steps stored in the memory on the depth sensor; and

in response to the second request and without storing additional operation steps in the memory, performing the second mode of operation by executing the second sequence of operation steps and the one or more common operation steps to generate the one or more second depth measurements,

wherein the first depth sensing mode of operation or the second depth sensing mode of operation is a long-range and low frame rate mode of operation.

17 . The method of claim 16 , wherein the long-range and low frame rate mode of operation is used for sensing depths at ranges from about 2-4 m with frame rates less than about 20 Hz.

18 . The method of claim 16 , wherein the depth sensor is a time-of-flight camera.

19 . A method for operating a depth sensor, the method comprising:

receiving, by the depth sensor, a first request for one or more first depth measurements according to a first depth sensing mode of operation of the depth sensor, wherein the first depth sensing mode of operation is defined by a first sequence of operation steps stored in memory on the depth sensor, the first sequence of operation steps including one or more common operation steps;

in response to the first request, performing the first mode of operation by executing the first sequence of operation steps to generate the one or more first depth measurements;

receiving, by the depth sensor, a second request for one or more second depth measurements according to a second depth sensing mode of operation of the depth sensor, wherein the second depth sensing mode of operation is defined by a combination of the one or more common operations steps and a second sequence of operation steps stored in the memory on the depth sensor; and

in response to the second request and without storing additional operation steps in the memory, performing the second mode of operation by executing the second sequence of operation steps and the one or more common operation steps to generate the one or more second depth measurements,

wherein:

the first depth sensing mode of operation is a short-range and high frame rate mode of operation;

the second depth sensing mode of operation is a short-range and low frame rate mode of operation;

the one or more common operation steps include steps to: i) capture a short-range intensity sub-frame using a short exposure, and ii) capture four phase sub-frames;

the first sequence of operation steps includes a first delay; and

the second sequence of operation steps includes a second delay that is longer than the first delay.

20 . A method for operating a depth sensor, the method comprising:

receiving, by the depth sensor, a first request for one or more first depth measurements according to a first depth sensing mode of operation of the depth sensor, wherein the first depth sensing mode of operation is defined by a first sequence of operation steps stored in memory on the depth sensor, the first sequence of operation steps including one or more common operation steps;

in response to the first request, performing the first mode of operation by executing the first sequence of operation steps to generate the one or more first depth measurements;

receiving, by the depth sensor, a second request for one or more second depth measurements according to a second depth sensing mode of operation of the depth sensor, wherein the second depth sensing mode of operation is defined by a combination of the one or more common operations steps and a second sequence of operation steps stored in the memory on the depth sensor; and

in response to the second request and without storing additional operation steps in the memory, performing the second mode of operation by executing the second sequence of operation steps and the one or more common operation steps to generate the one or more second depth measurements,

wherein:

the first depth sensing mode of operation is a long-range and high frame rate mode of operation;

the second depth sensing mode of operation is a long-range and low frame rate mode of operation;

the one or more common operation steps include steps to: i) capture a long-range intensity sub-frame using a long exposure, ii) capture four phase sub-frames for a first illumination modulation frequency, and iii) capture four phase sub-frames for a second illumination modulation frequency;

the first sequence of operation steps includes a first delay; and

the second sequence of operation steps includes a second delay that is longer than the first delay.

Assignments (2)
SECURITY INTEREST Recorded Oct 29, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073438/0463 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 16, 2024
From: SMITH, BRIAN KEITH; SHEE, KOON KEONG; LINK, GREGORY MICHAEL
To: MAGIC LEAP, INC.
Reel/Frame 069598/0860 →
Continuity (6)
Continuation 18453762 · Aug 22, 2023
Continuation 17690956 · Mar 9, 2022
Continuation 16573891 · Sep 17, 2019
Continuation 15925577 · Mar 19, 2018
Provisional Application 62474503 · Mar 21, 2017
Related Publication 20250088739A1 · Mar 13, 2025
References Cited (114)
US 4628312A · Hwang et al. · 1986 [cited by applicant]
US 6850221B1 · Tickle · 2005 [cited by applicant]
US D514570S · Ohta · 2006 [cited by applicant]
US 7852461B2 · Yahav · 2010 [cited by examiner]
US 8681255B2 · Katz · 2014 [cited by examiner]
US 8950867B2 · Macnamara · 2015 [cited by applicant]
US 9081426B2 · Armstrong · 2015 [cited by examiner]
US 9142019B2 · Lee · 2015 [cited by examiner]
US D752529S · Loretan et al. · 2016 [cited by applicant]
US 9307168B2 · Sambonsugi · 2016 [cited by applicant]
US 9310559B2 · Macnamara · 2016 [cited by applicant]
US D758367S · Natsume · 2016 [cited by applicant]
US D759657S · Kujawski et al. · 2016 [cited by applicant]
US 9389069B2 · Bloom et al. · 2016 [cited by applicant]
US 9407837B2 · Lee · 2016 [cited by examiner]
US 9432916B2 · Min et al. · 2016 [cited by applicant]
US 9683834B2 · Hsin et al. · 2017 [cited by applicant]
US D794288S · Beers et al. · 2017 [cited by applicant]
US D805734S · Fisher et al. · 2017 [cited by applicant]
US 9967535B2 · Laroia et al. · 2018 [cited by applicant]
US 10455153B2 · Smith et al. · 2019 [cited by applicant]
US 10477157B1 · Shahdi et al. · 2019 [cited by applicant]
US 11303809B2 · Smith et al. · 2022 [cited by applicant]
US 11778318B2 · Smith et al. · 2023 [cited by applicant]
US 12192623B2 · Smith et al. · 2025 [cited by applicant]
US 20010046317A1 · Kamon et al. · 2001 [cited by applicant]
US 20060028436A1 · Armstrong · 2006 [cited by applicant]
US 20070081123A1 · Lewis · 2007 [cited by applicant]
US 20080242931A1 · Nishino · 2008 [cited by applicant]
US 20090128833A1 · Yahav · 2009 [cited by applicant]
US 20100246327A1 · Yoshida · 2010 [cited by applicant]
US 20110205042A1 · Takemura et al. · 2011 [cited by applicant]
US 20120033957A1 · Itoh · 2012 [cited by applicant]
US 20120056982A1 · Katz et al. · 2012 [cited by applicant]
US 20120075534A1 · Katz et al. · 2012 [cited by applicant]
US 20120127062A1 · Bar-Zeev et al. · 2012 [cited by applicant]
US 20120162549A1 · Gao et al. · 2012 [cited by applicant]
US 20120232397A1 · Ohshima · 2012 [cited by examiner]
US 20130082922A1 · Miller · 2013 [cited by applicant]
US 20130117377A1 · Miller · 2013 [cited by applicant]
US 20130125027A1 · Abovitz · 2013 [cited by applicant]
US 20130126716A1 · Lee et al. · 2013 [cited by applicant]
US 20130208234A1 · Lewis · 2013 [cited by applicant]
US 20130242262A1 · Lewis · 2013 [cited by applicant]
US 20130271629A1 · Sambonsugi · 2013 [cited by examiner]
US 20140071539A1 · Gao · 2014 [cited by applicant]
US 20140177023A1 · Gao et al. · 2014 [cited by applicant]
US 20140218468A1 · Gao et al. · 2014 [cited by applicant]
US 20140240492A1 · Lee · 2014 [cited by examiner]
US 20140241614A1 · Lee · 2014 [cited by applicant]
US 20140267420A1 · Schowengerdt et al. · 2014 [cited by applicant]
US 20140306866A1 · Miller et al. · 2014 [cited by applicant]
US 20150016777A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150103306A1 · Kaji et al. · 2015 [cited by applicant]
US 20150170396A1 · Kornmann · 2015 [cited by examiner]
US 20150178939A1 · Bradski et al. · 2015 [cited by applicant]
US 20150205126A1 · Schowengerdt · 2015 [cited by applicant]
US 20150222883A1 · Welch · 2015 [cited by applicant]
US 20150222884A1 · Cheng · 2015 [cited by applicant]
US 20150268415A1 · Schowengerdt et al. · 2015 [cited by applicant]
US 20150281671A1 · Bloom et al. · 2015 [cited by applicant]
US 20150302652A1 · Miller et al. · 2015 [cited by applicant]
US 20150309263A2 · Abovitz et al. · 2015 [cited by applicant]
US 20150326570A1 · Publicover et al. · 2015 [cited by applicant]
US 20150346490A1 · TeKolste et al. · 2015 [cited by applicant]
US 20150346495A1 · Welch et al. · 2015 [cited by applicant]
US 20160011419A1 · Gao · 2016 [cited by applicant]
US 20160026253A1 · Bradski et al. · 2016 [cited by applicant]
US 20160140760A1 · Bowden et al. · 2016 [cited by applicant]
US 20160173892A1 · Park et al. · 2016 [cited by applicant]
US 20160180853A1 · VanLund et al. · 2016 [cited by applicant]
US 20160266647A1 · Leheup · 2016 [cited by applicant]
US 20160309133A1 · Laroia · 2016 [cited by examiner]
US 20170005456A1 · Broudno · 2017 [cited by examiner]
US 20170005465A1 · Wyland et al. · 2017 [cited by applicant]
US 20170148168A1 · Lindner et al. · 2017 [cited by applicant]
US 20180011194A1 · Masuda · 2018 [cited by examiner]
US 20180045513A1 · Kitamura et al. · 2018 [cited by applicant]
US 20180115730A1 · Velichko et al. · 2018 [cited by applicant]
US 20180278843A1 · Smith et al. · 2018 [cited by applicant]
US 20200014849A1 · Smith et al. · 2020 [cited by applicant]
US 20220201210A1 · Smith et al. · 2022 [cited by applicant]
US 20230396881A1 · Smith et al. · 2023 [cited by applicant]
CN 103518179A · 2014 [cited by applicant]
CN 105357511B · 2016 [cited by applicant]
JP H053568A · 1993 [cited by applicant]
JP H0553568A · 1993 [cited by applicant]
JP 2004157061A · 2004 [cited by applicant]
JP 2008183049A · 2008 [cited by applicant]
JP 2010085277B · 2010 [cited by applicant]
JP 2011007616A · 2011 [cited by applicant]
JP 2013123172A · 2013 [cited by applicant]
JP 2013159129A · 2013 [cited by applicant]
JP 2014509417A · 2014 [cited by applicant]
JP 2016503629A · 2016 [cited by applicant]
JP 2016213786A · 2016 [cited by applicant]
WO 2018175344A1 · 2018 [cited by applicant]
JP2023-197268 Office Action mailed Feb. 3, 2025. [cited by applicant]
CN2024110289276 Office Action dated Jun. 20, 2025. [cited by applicant]
Adeva, et al., “Efficient Architecture for Soft-Input Soft-Output Sphere Detection with Perfect Node Enumeration,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. XX, No. X, XXX, Jun. 8, 2015. [cited by applicant]
ARToolKit: https://web.archive.org/web/20051013062315/http://www.hitl.washington.edu:80/artoolkit/documentation/hardware.htm, archived Oct. 13, 2005. [cited by applicant]
Azuma, “A Survey of Augmented Reality,” Teleoperators and Virtual Environments 6, 4 (Aug. 1997), pp. 355-385. https://web.archive.org/web/20010604100006/http://www.cs.unc.edu/ azuma/ARpresence.pdf. [cited by applicant]
Azuma, “Predictive Tracking for Augmented Realty,” TR95-007, Department of Computer Science, UNC-Chapel Hill, NC, Feb. 1995. [cited by applicant]
Bimber, et al., “Spatial Augmented Reality—Merging Real and Virtual Worlds,” 2005 https://web.media.mit.edu/raskar/book/BimberRaskarAugmentedRealityBook.pdf. [cited by applicant]
CN2021108573590 Office Action dated Feb. 4, 2024 (entire contents). [cited by applicant]
CN2024110289276 Decision of Rejection dated Nov. 20, 2025 (entire contents). [cited by applicant]
EP22157426.2 Extended European Search Report dated Aug. 19, 2022. [cited by applicant]
IN202248043664 Examination Report dated Feb. 16, 2023. [cited by applicant]
International Preliminary Report on Patentability for PCT Application No. PCT/US18/23179, dated Sep. 24, 2019. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US18/23179, dated Jul. 6, 2018. [cited by applicant]
Invitation to Pay Additional Fees And, Where Applicable, Protest Fee for PCT Application No. PCT/US18/23179, mailed May 15, 2018. [cited by applicant]
Jacob, “Eye Tracking in Advanced Interface Design,” Human-Computer Interaction Lab Naval Research Laboratory, Washington, D.C. / paper/ in Virtual Environments and Advanced Interface Design, ed. by W. Barfield and T.A. … [cited by applicant]
JP2022-184776 Office Action mailed Feb. 21, 2024 (entire contents). [cited by applicant]
Tanriverdi and Jacob, “Interacting With Eye Movements in Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA—paper/Proc. AMC CHI 2000 Human Factors in Computin… [cited by applicant]