IP Library › Granted Patent US 12,190,486
Granted Patent B2
US 12,190,486 · App. 18/487,794 · Granted Jan 7, 2025

Virtual, augmented, and mixed reality systems and methods

Inventors: Jose Felix Rodriguez (Hialeah, FL); Ricardo Martinez Perez (Plantation, FL)
Assignee: Magic Leap, Inc.
G06T5/80G06F1/3203G06F3/14G06T3/18G06T3/4007G06T2207/10024G06T2210/44G09G2330/021G09G2370/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,486
App. No.
18/487,794
Granted
Jan 7, 2025
Kind
B2
Abstract

A virtual, augmented, or mixed reality display system includes a display configured to display virtual, augmented, or mixed reality image data, the display including one or more optical components which introduce optical distortions or aberrations to the image data. The system also includes a display controller configured to provide the image data to the display. The display controller includes memory for storing optical distortion correction information, and one or more processing elements to at least partially correct the image data for the optical distortions or aberrations using the optical distortion correction information.

Claims (24)

1. A virtual, augmented, or mixed reality display system comprising:

a display configured to display digital image data for a plurality of depth planes, the display comprising a first optical path to display image data corresponding to a first depth plane and an at least partially distinct second optical path to display image data corresponding to a second depth plane; and

a display controller configured to blend the image data corresponding to the first depth plane with the image data corresponding to the second depth plane such that when the blended image data is displayed by the display it appears to correspond to a virtual depth plane located between the first depth plane and the second depth plane.

2. The system of claim 1 , wherein the display controller is configured to blend the image data corresponding to the first depth plane with the image data corresponding to the second depth plane by calculating a weighted combination of the image data corresponding to the first depth plane and the image data corresponding to the second depth plane.

3. The system of claim 2 , wherein the display controller is configured to determine the weighted combination based on virtual depth plane indicator information embedded in the digital image data, the virtual depth plane indicator data specifying one of a plurality of possible virtual depth planes.

4. The system of claim 3 , wherein the virtual depth plane indicator data is embedded in pixel values of the digital image data.

5. The system of claim 4 , wherein the digital image data comprises a plurality of color values for each pixel, the plurality of least significant bits of the color values being the virtual depth plane indicator data.

6. The system of claim 4 , wherein the virtual depth plane indicator data is used to access a blending weight value associated with the virtual depth plane specified by the virtual depth plane indicator data.

7. The system of claim 6 , wherein blending weight values for each virtual depth plane are stored in a lookup table.

8. The system of claim 7 , wherein one or more lookup tables are provided for each depth plane of the digital image data.

9. The system of claim 1 , wherein the display controller is configured to blend all pixels of image data corresponding to the first depth plane and all pixels of image data corresponding to the second depth plane to a same virtual depth plane.

10. The system of claim 1 , wherein the display controller is configured to blend different pixels of image data corresponding to the first depth plane to different virtual depth planes.

11. A method in a virtual, augmented, or mixed reality display system, the method comprising:

providing digital image data for a plurality of depth planes to be shown on a display, the display comprising a first optical path to display image data corresponding to a first depth plane and an at least partially distinct second optical path to display image data corresponding to a second depth plane; and

blending the image data corresponding to the first depth plane with the image data corresponding to the second depth plane such that when the blended image data is displayed by the display it appears to correspond to a virtual depth plane located between the first depth plane and the second depth plane.

12. The method of claim 11 , further comprising blending the image data corresponding to the first depth plane with the image data corresponding to the second depth plane by calculating a weighted combination of the image data corresponding to the first depth plane and the image data corresponding to the second depth plane.

13. The method of claim 12 , further comprising determining the weighted combination based on virtual depth plane indicator information embedded in the digital image data, the virtual depth plane indicator data specifying one of a plurality of possible virtual depth planes.

14. The method of claim 13 , wherein the virtual depth plane indicator data is embedded in pixel values of the digital image data.

15. The method of claim 14 , wherein the digital image data comprises a plurality of color values for each pixel, the plurality of least significant bits of the color values being the virtual depth plane indicator data.

16. The method of claim 14 , further comprising using the virtual depth plane indicator data to access a blending weight value associated with the virtual depth plane specified by the virtual depth plane indicator data.

17. The method of claim 16 , further comprising storing blending weight values for each virtual depth plane in a lookup table.

18. The method of claim 17 , further comprising providing one or more lookup tables for each depth plane of the digital image data.

19. The method of claim 11 , further comprising blending all pixels of image data corresponding to the first depth plane and all pixels of image data corresponding to the second depth plane to a same virtual depth plane.

20. The method of claim 11 , further comprising blending different pixels of image data corresponding to the first depth plane to different virtual depth planes.

Assignments (3)
SECURITY INTEREST Recorded Oct 31, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073422/0549 →
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073031/0206 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 3, 2024
From: RODRIGUEZ, JOSE FELIX; PEREZ, RICARDO MARTINEZ
To: MAGIC LEAP, INC.
Reel/Frame 067911/0893 →
Continuity (8)
Continuation 17476392 · Sep 15, 2021
Continuation 16926484 · Jul 10, 2020
Continuation 16697711 · Nov 27, 2019
Continuation 15683677 · Aug 22, 2017
Provisional Application 62462279 · Feb 22, 2017
Provisional Application 62377829 · Aug 22, 2016
Provisional Application 62377804 · Aug 22, 2016
Related Publication 20240046433A1 · Feb 8, 2024
References Cited (170)
US 5583656A · Gandhi et al. · 1996 [cited by applicant]
US 5760636A · Noble et al. · 1998 [cited by applicant]
US 6988214B1 · Verdun · 2006 [cited by applicant]
US 9791700B2 · Schowengerdt · 2017 [cited by applicant]
US 10509466B1 · Miller et al. · 2019 [cited by applicant]
US 10529063B2 · Rodriguez et al. · 2020 [cited by applicant]
US 10748259B2 · Rodriguez et al. · 2020 [cited by applicant]
US 20030193566A1 · Matsuda et al. · 2003 [cited by applicant]
US 20040150888A1 · Domjan et al. · 2004 [cited by applicant]
US 20090059096A1 · Yamamoto et al. · 2009 [cited by applicant]
US 20090160978A1 · Getman et al. · 2009 [cited by applicant]
US 20090179824A1 · Tsujimoto · 2009 [cited by applicant]
US 20090295794A1 · Fan et al. · 2009 [cited by applicant]
US 20100091027A1 · Oyama et al. · 2010 [cited by applicant]
US 20100091031A1 · Tsujimoto · 2010 [cited by applicant]
US 20110075257A1 · Hua et al. · 2011 [cited by applicant]
US 20120159218A1 · Vangala et al. · 2012 [cited by applicant]
US 20120194419A1 · Osterhout et al. · 2012 [cited by applicant]
US 20130339757A1 · Reddy et al. · 2013 [cited by applicant]
US 20140009368A1 · Hirota · 2014 [cited by applicant]
US 20140184640A1 · Putraya · 2014 [cited by examiner]
US 20140376773A1 · Holz · 2014 [cited by applicant]
US 20150002542A1 · Chan et al. · 2015 [cited by applicant]
US 20150009416A1 · Tamayama et al. · 2015 [cited by applicant]
US 20150237323A1 · Du et al. · 2015 [cited by applicant]
US 20150331480A1 · Zhou et al. · 2015 [cited by applicant]
US 20150341607A1 · Kobayashi et al. · 2015 [cited by applicant]
US 20150379697A1 · Pohl · 2015 [cited by applicant]
US 20160026253A1 · Bradski et al. · 2016 [cited by applicant]
US 20160037073A1 · Mills et al. · 2016 [cited by applicant]
US 20160191815A1 · Annau · 2016 [cited by applicant]
US 20160314564A1 · Jones · 2016 [cited by applicant]
US 20160327789A1 · Klug et al. · 2016 [cited by applicant]
US 20170053450A1 · Rodriguez et al. · 2017 [cited by applicant]
US 20170115489A1 · Hu · 2017 [cited by applicant]
US 20170169612A1 · Cashen et al. · 2017 [cited by applicant]
US 20170206689A1 · Eo · 2017 [cited by applicant]
US 20170287447A1 · Barry · 2017 [cited by examiner]
US 20170329136A1 · Bates · 2017 [cited by applicant]
US 20200209638A1 · Osmanis et al. · 2020 [cited by applicant]
AU 2022201611B2 · 2022 [cited by applicant]
AU 2023201212B2 · 2024 [cited by applicant]
CN 104115491 · 2014 [cited by applicant]
JP 3022558 · 2000 [cited by applicant]
JP 2000261832 · 2000 [cited by applicant]
JP 2001320738 · 2001 [cited by applicant]
JP 2004109994 · 2004 [cited by applicant]
JP 2004236191 · 2004 [cited by applicant]
JP 2004274642 · 2004 [cited by applicant]
JP 2007166277 · 2007 [cited by applicant]
JP 2009105593 · 2009 [cited by applicant]
JP 2010016669 · 2010 [cited by applicant]
JP 2012033038 · 2012 [cited by applicant]
JP 2013061442 · 2013 [cited by applicant]
JP 2013104987 · 2013 [cited by applicant]
JP 201413968 · 2014 [cited by applicant]
KR 1020140122196 · 2014 [cited by applicant]
KR 1020150105264 · 2015 [cited by applicant]
KR 1020160033763 · 2016 [cited by applicant]
KR 1020160149252 · 2016 [cited by applicant]
WO WO2013125138 · 2013 [cited by applicant]
WO WO2015081313 · 2015 [cited by applicant]
WO WO2015164397 · 2015 [cited by applicant]
WO WO2016014608 · 2016 [cited by applicant]
WO WO2016032973 · 2016 [cited by applicant]
WO WO2016054092 · 2016 [cited by applicant]
Foreign Response for CA Patent Appln. No. 3034384 dated Jan. 19, 2023. [cited by applicant]
Foreign Response for JP Patent Appln. No. 2022-162258 dated Feb. 15, 2024. [cited by applicant]
Foreign OA for JP Patent Appln. No. 2022-162258 dated Nov. 17, 2023 (with English translation). [cited by applicant]
Foreign Patent Grant for IN Patent Appln. No. 201947009233 dated Dec. 7, 2023 (in English). [cited by applicant]
Extended European Search Report for European Appln. No. 17844304.0 dated Aug. 9, 2019 (7 pages). [cited by applicant]
“A Method of Computational Correction for Optical Distortion in Head-Mounted Displays”, XP055288742, Dept. of Computer Science, University of North Carolina, dated Jan. 1, 1993. [cited by applicant]
PCT International Search Report and Written Opinion for PCT/US2017/048069, Applicant Magic Leap, Inc., mailed Jan. 18, 2018, 21 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/697,711, dated Jan. 13, 2020. [cited by applicant]
Response to Extended European Search Report for European Appln. No. 17844304.0 filed Mar. 5, 2020 (76 pages). [cited by applicant]
Notice of Allowance dated Sep. 26, 2019 for U.S. Appl. No. 15/683,677. [cited by applicant]
Amendment Response to Non-Final Office Action dated Apr. 10, 2020 for U.S. Appl. No. 16/697,711. [cited by applicant]
Foreign Notice of Allowance for IL Patent Appln. No. 264666 dated May 12, 2020 (no translation available). [cited by applicant]
1st Examination Report for European Appln. No. 17844304.0 dated Jun. 5, 2020 (4 pages). [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/926,484, dated Aug. 5, 2020. [cited by applicant]
Response to Non-Final Office Action for U.S. Appl. No. 16/926,484, filed Nov. 5, 2020. [cited by applicant]
Foreign OA for JP Patent Application No. 2019-510409 dated Nov. 10, 2020. [cited by applicant]
Foreign OA for KR Patent Appln. No. 10-2019-7008251 dated Nov. 19, 2020. [cited by applicant]
Foreign Response for EP Patent Appln. No. 17844304.0 dated Dec. 15, 2020. [cited by applicant]
Final Office Action for U.S. Appl. No. 16/926,484 dated Feb. 4, 2021. [cited by applicant]
Foreign Response for KR Patent Appln. No. 10-2019-7008251 dated Jan. 15, 2021. [cited by applicant]
Foreign Notice of Allowance for KR Patent Appln. No. 10-2019-7008251 dated Feb. 1, 2021. [cited by applicant]
Foreign First Exam Report for AU Patent Appln. No. 2017317600 dated Jan. 19, 2021. [cited by applicant]
Foreign NOA for JP Patent Appln. No. 2019-510409 dated Apr. 6, 2021. [cited by applicant]
Foreign OA for IL Patent Appln. No. 276547 dated Feb. 14, 2021. [cited by applicant]
2nd Exam Report for EP Patent Appln. No. 17844304.0 dated Mar. 26, 2021. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/926,484 dated May 24, 2021. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/926,484 dated Jun. 10, 2021. [cited by applicant]
Foreign NOA for KR Patent Appln. No. 10-2021-7012256 dated Aug. 25, 2021. [cited by applicant]
Foreign Response for EP Patent Appln. No. 17844304.0 dated Oct. 5, 2021. [cited by applicant]
Foreign NOA for IL Patent Appln. No. 276547 dated Jul. 18, 2021. [cited by applicant]
Foreign Response for KR Patent Appln. No. 10-2021-7012256 dated Oct. 20, 2021. [cited by applicant]
Foreign NOA for KR Patent Appln. No. 10-2021-7012256 dated Oct. 29, 2021. [cited by applicant]
Foreign OA for JP Patent Appln. No. 2020-138439 dated Oct. 21, 2021. [cited by applicant]
Foreign OA for IN Patent Appln. No. 201947009233 dated Nov. 16, 2021. [cited by applicant]
Foreign Response for AU Patent Appln. No. 2017317600 dated Nov. 24, 2021. [cited by applicant]
Foreign Response for JP Patent Appln. No. 2020-138439 dated Jan. 13, 2022. [cited by applicant]
Foreign FOA for JP Patent Appln. No. 2020-138439 dated Jun. 7, 2022 with English translation. [cited by applicant]
Foreign Exam Report for NZ Patent Appln. No. 791441 dated Sep. 20, 2022. [cited by applicant]
Foreign Exam Report for NZ Patent Appln. No. 750575 dated Sep. 20, 2022. [cited by applicant]
Foreign Exam Report for NZ Patent Appln. No. 791444 dated Sep. 20, 2022. [cited by applicant]
Foreign Response for CN Patent Appln. No. 201780065160.6 dated Oct. 24, 2022. [cited by applicant]
Request for Reexamination of U.S. Pat. No. 10,748,259 under 35 U.S.C. § 302 and 37 C.F.R. § 1.510 dated Oct. 7, 2022. [cited by applicant]
Declaration of Dr. Alexander V. Sergienko Under 37 C.F.R. § 1.132 including Appendix A (List of Materials Considered) and Appendix B (Curriculum Vitae) dated Oct. 7, 2022. [cited by applicant]
Milgram, P. and Kishino, F., “A Taxonomy of Mixed Reality Visual Displays,” IEICE Transactions on Information and Systems, E77-D(12):1321-1329, Dec. 1994. [cited by applicant]
Azuma, R., “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments, 6(4):355-385, Aug. 1997. [cited by applicant]
Carmigniani, J., et al., “Augmented reality technologies, systems and applications,” Multimedia Tools and Applications, 51(1):341-377, Dec. 2010. [cited by applicant]
Sutherland, Ivan. “The ultimate display”, Proceedings of IFIP Congress, pp. 506-508, 1965. [cited by applicant]
Feiner, S., “The Importance of Being Mobile: Some Social Consequences of Wearable Augmented Reality Systems,” Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99), Oct. 1999, pp. 145-148. [cited by applicant]
Chung, J., et al., “Exploring virtual worlds with head-mounted displays.” In Three-Dimensional Visualization and Display Technologies, vol. 1083, pp. 42-52. International Society for Optics and Photonics, 1989. [cited by applicant]
Burke, S., “The History of Virtual Reality & The Future: Rift, Omni, STEM, castAR”, Oct. 2013. [cited by applicant]
Rolland, J. P., et al., “Comparison of optical and video see-through, head-mounted displays”, Proc. SPIE 2351, Telemanipulator and Telepresence Technologies, pp. 293-307, Jan. 1994. [cited by applicant]
Azuma, R., “Recent advances in augmented reality,” in IEEE Computer Graphics and Applications, vol. 21, No. 6, pp. 34-47, Nov.-Dec. 2001. [cited by applicant]
Holliman, N.S., “Three-Dimensional Displays: A Review and Applications Analysis,” in IEEE Transactions on Broadcasting, vol. 57, No. 2, pp. 362-371, Jun. 2011. [cited by applicant]
Sony Releases the Transparent Lens Eyewear “SmartEyeglass Developer Edition”, Feb. 2015. [cited by applicant]
Wearable Tech 2013: Vuzix reveals Waveguide optics engine for smart glasses, Dec. 2013. [cited by applicant]
Rolland, J.P. and Hopkins, T, “A method of computational correction for optical distortion in head-mounted displays”, Technical Report TR93-045, Dept. of Computer Science, University of North Carolina at Chapel Hill, 19… [cited by applicant]
B. A. Watson and L. F. Hodges, “Using texture maps to correct for optical distortion in head-mounted displays,” Proceedings Virtual Reality Annual International Symposium '95, 1995, pp. 172-178. [cited by applicant]
File History of U.S. Pat. No. 10,748,259. [cited by applicant]
Claim Chart 1 for U.S. Pat. No. 10,748,259 dated Oct. 7, 2022. [cited by applicant]
Claim Chart 2 for U.S. Pat. No. 10,748,259 dated Oct. 7, 2022. [cited by applicant]
Claim Chart 3 for U.S. Pat. No. 10,748,259 dated Oct. 7, 2022. [cited by applicant]
Claim Chart 4 for U.S. Pat. No. 10,748,259 dated Oct. 7, 2022. [cited by applicant]
Claim Chart 5 for U.S. Pat. No. 10,748,259 dated Oct. 7, 2022. [cited by applicant]
Claim Chart 6 for U.S. Pat. No. 10,748,259 dated Oct. 7, 2022. [cited by applicant]
Request for Reexamination of U.S. Pat. No. 10,529,063 under 35 U.S.C. § 302 and 37 C.F.R. § 1.510 dated Oct. 14, 2022. [cited by applicant]
Declaration of Dr. Alexander V. Sergienko Under 37 C.F.R. § 1.132 including Appendix A (List of Materials Considered) and Appendix B (Curriculum Vitae) and Incorporated Claim Charts CC-1 to CC-3 dated Oct. 14, 2022. [cited by applicant]
File History of U.S. Pat. No. 10,529,063. [cited by applicant]
Claim Chart 1 for U.S. Pat. No. 10,529,063 dated Oct. 14, 2022. [cited by applicant]
Claim Chart 2 for U.S. Pat. No. 10,529,063 dated Oct. 14, 2022. [cited by applicant]
Claim Chart 3 for U.S. Pat. No. 10,529,063 dated Oct. 14, 2022. [cited by applicant]
Foreign OA for CN Patent Appln. No. 201780065160.6 dated Nov. 22, 2022 (with English translation). [cited by applicant]
Notice of Acceptance for AU Patent Appln. No. 2022201611 dated Dec. 1, 2022. [cited by applicant]
Foreign OA for KR Patent Appln. No. 10-2022-7002570 dated Feb. 2, 2023. [cited by applicant]
Foreign Response for CN Patent Appln. No. 201780065160.6 dated Mar. 21, 2023. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/476,392 dated Mar. 31, 2023. [cited by applicant]
Foreign Response for KR Patent Appln. No. 10-2022-7002570 dated Mar. 29, 2023. [cited by applicant]
Foreign OA for JP Patent Appln. No. 2020-138439 dated Apr. 3, 2023. [cited by applicant]
Foreign NOA for KR Patent Appln. No. 10-2022-7002570 dated May 12, 2023. [cited by applicant]
Foreign Decision to Grant for EP Patent Appln. No. 17844304.0 dated Jun. 2, 2023. [cited by applicant]
Amendment Response to NFOA for U.S. Appl. No. 17/476,392 dated Jun. 27, 2023. [cited by applicant]
Amendment Response to OA for Reexamination U.S. Appl. No. 90/019,122 dated Jun. 30, 2023. [cited by applicant]
Foreign Response for CN Patent Appln. No. 201780065160.6 dated Jul. 7, 2023. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/476,392 dated Jul. 17, 2023. [cited by applicant]
Foreign Exam Report for NZ Patent Appln. No. 791441 dated Jul. 24, 2023. [cited by applicant]
Foreign OA for CN Patent Appln. No. 201780065160.6 dated Aug. 10, 2023. [cited by applicant]
Amendment Response for Reexamination U.S. Appl. No. 90/019,118 dated Aug. 24, 2023. [cited by applicant]
Foreign OA for KR Patent Appln. No. 10-2023-7026979 dated Aug. 24, 2023. [cited by applicant]
Foreign Response for KR Patent Appln. No. 10-2023-7026979 dated Aug. 31, 2023. [cited by applicant]
Final Office Action for Reexamination U.S. Appl. No. 90/019,122 dated Aug. 29, 2023. [cited by applicant]
Foreign NOA for IL Patent Appln. No. 287380 dated Jun. 28, 2023 in English. [cited by applicant]
Notice of Intent to Issue Ex Parte Reexamination Certificate for Reexamination U.S. Appl. No. 90/019,118 dated Sep. 14, 2023. [cited by applicant]
Foreign NOA for NZ Patent Appln. No. 791441 dated Sep. 20, 2023. [cited by applicant]
Foreign Exam Report for EP Patent Appln. No. 3034384 dated Sep. 26, 2023. [cited by applicant]
Amendment Response to FOA for Reexamination U.S. Appl. No. 90/019,122 dated Oct. 26, 2023. [cited by applicant]
Extended European Search Report for EP Patent Appln. No. 23174743.7 dated Sep. 22, 2023. [cited by applicant]
Foreign Exam Report for AU Patent Appln. No. 2023201212 dated Oct. 31, 2023. [cited by applicant]
Foreign NOA for KR Patent Appln. No. 10-2023-7026979 dated Nov. 1, 2023. [cited by applicant]
Foreign NOA for CN Patent Appln. No. 201780065160.6 dated Dec. 26, 2023. [cited by applicant]
Foreign Response for EP Patent Appln. No. 23174743.7 dated Apr. 25, 2024. [cited by applicant]
Foreign NOA for JP Patent Appln. No. 2022-162258 dated May 21, 2024. [cited by applicant]
Foreign NOA for AU Patent Appln. No. 2023201212 dated Jun. 17, 2024. [cited by applicant]
Foreign Exam Report Response for AU Patent Appln. No. 2023201212 dated Jun. 6, 2024. [cited by applicant]
Foreign OA for KR Patent Appln. No. 10-2024-7003927 dated Aug. 13, 2024. [cited by applicant]
Foreign OA Response for KR Patent Appln. No. 10-2024-7003927 dated Oct. 15, 2024. [cited by applicant]