IP Library › Granted Patent US 12,657,449
Granted Patent B2
US 12,657,449 · App. 17/629,346 · Granted Jun 16, 2026

Optical systems and methods using broadband diffractive neural networks

Inventors: Aydogan Ozcan (Los Angeles, CA); Yi Luo (Los Angeles, CA); Deniz Mengu (Los Angeles, CA); Yair Rivenson (Los Angeles, CA)
Assignee: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
G06N3/067G06N3/084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,449
App. No.
17/629,346
Granted
Jun 16, 2026
Kind
B2
Abstract

A broadband diffractive optical neural network simultaneously processes a continuum of wavelengths generated by a temporally-incoherent broadband source to all-optically perform a specific task learned using network learning. The optical neural network design was verified by designing, fabricating and testing seven different multi-layer, diffractive optical systems that transform the optical wavefront generated by a broadband THz pulse to realize (1) a series of tunable, single passband as well as dual passband spectral filters, and (2) spatially-controlled wavelength de-multiplexing. Merging the native or engineered dispersion of various material systems with a deep learning-based design, broadband diffractive optical neural networks help engineer light-matter interaction in 3D, diverging from intuitive and analytical design methods to create task-specific optical components that can all-optically perform deterministic tasks or statistical inference for optical machine learning. The optical neural network may be implemented as a reflective optical neural network.

Claims (27)

1 . An optical neural network device for processing a broadband input optical signal from a broadband light source comprising:

a plurality of optically transmissive and/or reflective substrate layers arranged in an optical path, each of the plurality of optically transmissive and/or reflective substrate layers comprising a plurality of physical features formed on or within the plurality of optically transmissive or reflective substrate layers and having different complex-valued transmission and/or reflection coefficients as a function of the lateral coordinates across each substrate layer, wherein the plurality of optically transmissive and/or reflective substrate layers and the plurality of physical features thereon collectively define a trained mapping function between the broadband input optical signal to the plurality of optically transmissive and/or reflective substrate layers and output optical signals created by optical diffraction through and/or optical reflection from the plurality of optically transmissive and/or reflective substrate layers;

one or more optical sensors configured to capture the output optical signals resulting from the plurality of optically transmissive and/or reflective substrate layers; and

wherein the plurality of optically transmissive and/or reflective substrate layers generate wavelength de-multiplexed optical signals for the one or more optical sensors.

2 . The optical neural network device of claim 1 , wherein the broadband input optical signal comprises a broadband terahertz (THz) source, broadband visible source, or a broadband infrared source.

3 . The optical neural network device of claim 1 , wherein the plurality of physical features of the plurality of optically transmissive and/or reflective substrate layers comprise regions of varied thicknesses or regions having different optical properties.

4 . The optical neural network device of claim 1 , wherein the plurality of physical features of the plurality of optically transmissive and/or reflective substrate layers comprise metamaterials and/or metasurfaces.

5 . The optical neural network device of claim 1 , wherein the plurality of optically transmissive and/or reflective substrate layers are positioned within and/or surrounded by vacuum, air, a gas, a liquid or a solid material.

6 . The optical neural network device of claim 1 , wherein the plurality of optically transmissive and/or reflective substrate layers comprise a nonlinear optical material.

7 . The optical neural network device of claim 1 , wherein the plurality of optically transmissive and/or reflective substrate layers comprises one or more physical substrate layers that comprise reconfigurable physical features that can change on demand as a function of time.

8 . A method of forming a multi-layer optical neural network for processing broadband light comprising:

training a software-based neural network to perform one or more specific optical functions for a multi-layer transmissive and/or reflective network having a plurality of optically diffractive physical features located in different locations in each of the layers of the transmissive and/or reflective network, wherein the training comprises feeding a plurality of different wavelengths of a broadband optical signal to the software-based neural network and computing at least one optical output of optical transmission and/or reflection through the multi-layer transmissive and/or reflective network using a wave propagation model and iteratively adjusting complex-valued transmission/reflection coefficients for each layer of the multi-layer transmissive and/or reflective network until optimized transmission/reflection coefficients are obtained or a certain time or epochs have elapsed; and

manufacturing or having manufactured a physical embodiment of the multi-layer transmissive and/or reflective network comprising a plurality of substrate layers having physical features that match the optimized transmission/reflection coefficients obtained by the trained neural network.

9 . The method of claim 8 , wherein the optimized transmission/reflective coefficients are obtained by error back-propagation.

10 . The method of claim 8 , wherein the optimized transmission/reflection coefficients are obtained by deep learning.

11 . The method of claim 8 , wherein the physical embodiment of the multi-layer transmissive and/or reflective network is manufactured by additive manufacturing or lithography.

12 . The method of claim 8 , wherein the plurality of optically transmissive and/or reflective substrate layers forming the network are positioned within and/or surrounded by vacuum, air, a gas, a liquid or a solid material.

13 . The method of claim 8 , wherein the physical embodiment of the multi-layer transmissive and/or reflective network comprises one or more physical substrate layers that comprise a nonlinear optical material.

14 . The method of claim 8 , wherein the physical embodiment of the multi-layer transmissive and/or reflective network comprises one or more physical substrate layers that comprise reconfigurable physical features that can change on demand as a function of time.

15 . The method of claim 8 , wherein the physical embodiment of the multi-layer transmissive and/or reflective network filters a particular wavelength, a plurality of particular wavelengths, a set of wavelengths, or sets of wavelengths within a wavelength range of the broadband light source.

16 . The method of claim 8 , wherein the physical embodiment of the multi-layer transmissive and/or reflective network generates wavelength de-multiplexed optical signals for one or more optical sensors.

17 . The method of claim 8 , wherein the broadband input optical signal comprises a broadband terahertz (THz) source, a broadband visible source, or a broadband infrared source.

18 . A method of processing a broadband input optical signal comprising:

providing an optical neural network device comprising:

a plurality of optically transmissive and/or reflective substrate layers arranged in an optical path, each of the plurality of optically transmissive and/or reflective substrate layers comprising a plurality of physical features formed on or within the plurality of optically transmissive or reflective substrate layers and having different complex-valued transmission and/or reflection coefficients as a function of the lateral coordinates across each substrate layer, wherein the plurality of optically transmissive and/or reflective substrate layers and the plurality of physical features thereon collectively define a trained mapping function between the broadband input optical signal to the plurality of optically transmissive and/or reflective substrate layers and output optical signals comprising wavelength de-multiplexed optical signals created by optical diffraction through and/or optical reflection from the plurality of optically transmissive and/or reflective substrate layers; and

one or more optical sensors configured to capture the one or more output wavelength de-multiplexed optical signals resulting from the plurality of optically transmissive and/or reflective substrate layers; and

inputting the broadband input optical signal into the optical neural network device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 21, 2022
From: OZCAN, AYDOGAN; LUO, YI; MENGU, DENIZ; RIVENSON, YAIR
To: THE REGENTS OF THE UNIVERSITY OF CALIFORNIA
Reel/Frame 058731/0202 →
Continuity (2)
Provisional Application 62900353 · Sep 13, 2019
Related Publication 20220253685A1 · Aug 11, 2022
References Cited (74)
US 5095459A · Ohta et al. · 1992 [cited by applicant]
US 5842191A · Stearns · 1998 [cited by applicant]
US 6445470B1 · Jenkins et al. · 2002 [cited by applicant]
US 6815683B2 · Federici et al. · 2004 [cited by applicant]
US 9459142B1 · Huseynov et al. · 2016 [cited by applicant]
US 10187171B2 · Doster · 2019 [cited by examiner]
US 20100209830A1 · Carcasi et al. · 2010 [cited by applicant]
US 20120262610A1 · Rissa et al. · 2012 [cited by applicant]
US 20140067741A1 · Katayama · 2014 [cited by examiner]
US 20140067742A1 · Katayama et al. · 2014 [cited by applicant]
US 20170351293A1 · Carolan et al. · 2017 [cited by applicant]
US 20180045953A1 · Fan et al. · 2018 [cited by applicant]
US 20180262291A1 · Doster et al. · 2018 [cited by applicant]
US 20210142170A1 · Ozcan et al. · 2021 [cited by applicant]
WO 2019200289A1 · 2019 [cited by applicant]
Russell Pollari, report on Lin, “All-Optical Machine Learning Using Diffractive Deep Neural Networks” Sep. 10, 2018 attached to the present Office action and see also youtube presentation at (https://www.youtube.com/wat… [cited by examiner]
PCT International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) for PCT/US2020/049950, Applicant: The Regents of the University of California, Form PCT/IB/326 and 373, dated Mar. 24, 2… [cited by applicant]
Bagherian, H. et al., On-Chip Optical Convolutional Neural Networks, arXiv:1808.03303v2 [cs.ET] Aug. 16, 2018. [cited by applicant]
Brunner, D. et al., Parallel photonic information processing at gigabyte per second data rates using transient states, Nature Communications, 4:1364, DOI: 10.1038/ncomms2368, www.nature.com/naturecommunications (2013). [cited by applicant]
Bueno, J. et al., Reinforcement Learning in a large scale photonic Recurrent Neural Network, arXiv:1711.05133v2 [cs.NE] Nov. 15, 2017. [cited by applicant]
Chakraborty, I. et al., Toward Fast Neural Computing using All-Photonic Phase Change Spiking Neurons, arXiv:1804.00267v2 [cs.ET] Aug. 28, 2018. [cited by applicant]
Chang, J. et al., Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification, Scientific Report (2018) 8:12324, DOI:10.1038/s41598-018-30619-y. [cited by applicant]
Chen, Y. et al., Deep Learning-Based Classification of Hyperspectral Data, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 7, No. 6, Jun. 2014. [cited by applicant]
Cignoni, P. et al., MeshLab: an Open-Source Mesh Processing Tool, Eurographics Italian Chapter Conference (2008). [cited by applicant]
Farhat, N. H. et al., Optical implementation of the Hopfield model, May 15, 1985, vol. 24, No. 10, Applied Optics, 1469-1475. [cited by applicant]
Girshick, R. et al., Rich feature hierarchies for accurate object detection and semantic segmentation Tech report (v5), arXiv:1311.2524v5 [cs.CV] Oct. 22, 2014. [cited by applicant]
Golik, P. et al., Cross-Entropy vs. Squared Error Training: a Theoretical and Experimental Comparison, Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, 52056 Aachen… [cited by applicant]
Grischkowsky, D. et al., Far-infrared time-domain spectroscopy with terahertz beams of dielectrics and semiconductors, J. Opt. Soc. Am. B/vol. 7, No. 10/Oct. 1990. [cited by applicant]
Hermans, M. etal., Trainable & Dynamic Computing: Error Backpropagation Through Physical Media, arXiv:1407.6637v1 [cs.NE] Jul. 24, 2014. [cited by applicant]
Hughes, T.W. et al., Training of photonic neural networks through in situ backpropagation, arXiv:1805.09943v1 [physics.optics] May 25, 2018. [cited by applicant]
Kazhdan, M. et al., Screened Poisson Surface Reconstruction, ACM Transactions on Graphics, vol. VV, No. N, Article XXX, Publication date: 2013. [cited by applicant]
Khorasaninejad, M. et al., Metalenses at visible wavelengths: Diffraction-limited focusing and subwavelength resolution imaging, sciencemag.org, Science, Jun. 3, 2016, vol. 352, Issue 6290, 1190-1194. [cited by applicant]
Kildishev, A. V. et al., Planar Photonics with Metasurfaces, Science 339, 1232009 (2013). DOI: 10.1126/science.1232009. [cited by applicant]
Kingma, D. P. et al., ADAM: a Method for Stochastic Optimization, arXiv:1412.6980v9 [cs.LG] Jan. 30, 2017. [cited by applicant]
Lecun, Y. et al., GradientBased Learning Applied to Document Recognition, Proc. of the IEEE, Nov. 1998 (46 pages). [cited by applicant]
Lin, X. et al., All-optical machine learning using diffractive deep neural networks, Science 361, 1004-1008 (2018), Sep. 7, 2018. [cited by applicant]
Psaltis, D. et al., Optical information processing based on an associative-memory model of neural nets with thresholding and feedback, Optics Letters, vol. 10, No. 2, Feb. 1985. [cited by applicant]
Psaltis, D. et al., Adaptive optical networks using photorefractive crystals, Applied Optics, vol. 27, No. 9, May 1, 1988. [cited by applicant]
Shastri, B. et al., Principles of Neuromorphic Photonics, arXiv:1801.00016v1 [cs.ET] Dec. 29, 2017. [cited by applicant]
Shen, Y. et al., Deep Learning with Coherent Nanophotonic Circuits, arXiv:1610.02365v1 [physics.optics] Oct. 7, 2016. [cited by applicant]
Srivastava, N. et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Journal of Machine Learning Research 15 (2014) 1929-1958 Submitted Nov. 2013; Published Jun. 2014. [cited by applicant]
Tang, Y. et al., Deep Learning using Linear Support Vector Machines, arXiv:1306.0239v4 [cs.LG] Feb. 21, 2015. [cited by applicant]
Trabelsi, C. et al., Deep Complex Networks, arXiv:1705.09792v4 [cs.NE] Feb. 25, 2018. [cited by applicant]
Wagner, K. et al., Multilayer optical learning networks, Dec. 1, 1987, vol. 26, No. 23, Applied Optics, 5061-5076. [cited by applicant]
Wan, Li et al., Regularization of Neural Networks using DropConnect, Proceedings of the 30 th International Conference on Machine Learning, Atlanta, Georgia, USA, 2013. [cited by applicant]
Wang, Z. et al., Image Quality Assessment: From Error Visibility to Structural Similarity, IEEE Transactions on Image Processing, vol. 13, No. 4, Apr. 2004. [cited by applicant]
Weverka, R. T. et al., Fully interconnected, two-dimensional neural arrays using wavelength-multiplexed volume holograms, Optics Letters, vol. 16, No. 11, Jun. 1, 1991. [cited by applicant]
Xiao, Y. et al., Nonlinear Metasurface Based on Giant Optical Kerr Response of Gold Quantum Wells, ACS Photonics, 5(5), May 1, 2018, DOI:10.1021/acsphotonics.7b01140. [cited by applicant]
Yin, X. et al., Artificial Kerr-type medium using metamaterials, Apr. 9, 2012, vol. 20, No. 8, Optics Express, 8543-8550. [cited by applicant]
PCT International Search Report for PCT/US2019/027275, Applicant: The Regents of the University of California, Form PCT/ISA/210 and 220, dated Aug. 19, 2019 (5 pages). [cited by applicant]
PCT Written Opinion of the International Search Authority for PCT/US2019/027275, Applicant: The Regents of the University of California, Form PCT/ISA/237, dated Aug. 19, 2019 (9 pages). [cited by applicant]
Javidi, B. et al., Optical implementation of neural networks for face recognition by the use of nonlinear joint transform correlators, Applied Optics, vol. 34, No. 20, Jul. 10, 1995, 3950-3962. [cited by applicant]
Jutamulia, S. et al., Overview of hybrid optical neural networks, Optics & Laser Technology, vol. 28, No. 2, pp. 59-72, 1996. [cited by applicant]
Psaltis, D. et al., Holography in artificial neural networks. Nature. 343, 325-330 (1990). [cited by applicant]
Soures, N. et al., Neuro-MMI: A Hybrid Photonic-Electronic Machine Learning Platform. In 2018 IEEE Photonics Society Summer Topical Meeting Series (SUM); 2018; pp. 187-188. [cited by applicant]
Yu, F. T. S., II Optical Neural Networks: Architecture, Design and Models. In Progress in Optics; Wolf, E., Ed.; Elsevier, 1993; vol. 32, pp. 61-144. [cited by applicant]
Yu, F. et al., Flat optics with designer metasurfaces. Nature Materials. 13, 139-150 (2014). [cited by applicant]
Caulfield, H.J. et al., Optical Neural Networks. Proc. IEEE 1989, 77 (10), 1573-1583. [cited by applicant]
PCT International Preliminary Report on Patentability (Chapter I of the Patent Cooperation Treaty) for PCT/US2019/027275, Applicant: The Regents of the University of California, Form PCT/IB/326 and 373, dated Oct. 22, 2… [cited by applicant]
Sara Abrahamsson et al., Multifocus microscopy with precise color multi-phase diffractive optics applied in functional neuronal imaging, Biomedical Optics Express, Mar. 1, 2016, vol. 7, No. 3, 855-869. [cited by applicant]
Francesco Aieta et al., Multiwavelength achromatic metasurfaces by dispersive phase compensation, sciencemag.org Science, Mar. 20, 2015, vol. 347, Issue 6228, 1342-1345. [cited by applicant]
Sourangsu Banerji et al., A Computational Design Framework for Efficient, Fabrication Error-Tolerant, Planar THz Diffractive Optical Elements, Scientific Reports, (2019) 9:5801, https://doi.org/10.1038/s41598-019-42243-… [cited by applicant]
Julie Chang et al., Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification, Scientific Reports, (2018) 8:12324; DOI:10.1038/s41598-018-30619-y. [cited by applicant]
Nicholas J. Karl et al., Frequency-division multiplexing in the terahertz range using a leaky-wave antenna, Nature Photonics, vol. 9, Nov. 2015, 717-721, www.nature.com/naturephotonics. [cited by applicant]
Diederik P. Kingma et al., Auto-Encoding Variational Bayes, arXiv: 1312.6114v10 [stat.ML] May 1, 2014. [cited by applicant]
Iñigo Liberal et al., Reconfigurable epsilon-near-zero metasurfaces via photonic doping, Nanophotonics 2018; 7(6): 1117-1127. [cited by applicant]
Jingxi Li et al., Class-specific Differential Detection in Diffractive Optical Neural Networks Improves Inference Accuracy (2019) (21 pages). [cited by applicant]
Xing Lin et al., All-optical machine learning using diffractive deep neural networks, Science 361, 1004-1008 (2018). [cited by applicant]
Yi Luo et al., Design of task-specific optical systems using broadband diffractive neural networks, Light: Science & Applications ( 2019) 8:112. [cited by applicant]
Shuming Wang et al., Broadband achromatic optical metasurface devices, Nature Communications, 8:187, DOI: 10.1038/s41467-017-00166-7 | www.nature.com/naturecommunications, (2017). [cited by applicant]
Yi Luo et al., Design of Task-Specific Optical Systems Using Broadband Diffractive Neural Networks, Light: Science & Applications. Dec. 2, 2019;8(1):1-4. [cited by applicant]
Deniz Mengu et al., Analysis of Diffractive Optical Neural Networks and Their Integration with Electronic Neural Networks, IEEE Journal of Selected Topics in Quantum Electronics, vol. 26, No. 1, Jan./Feb. 2020 (14 pages… [cited by applicant]
PCT International Search Report for PCT/US2020/049950, Applicant: The Regents of the University of California, Form PCT/ISA/210 and 220, dated Nov. 30, 2020 (4 pages). [cited by applicant]
PCT Written Opinion of the International Search Authority for PCT/US2020/049950, Applicant: The Regents of the University of California, Form PCT/ISA/237, dated Nov. 30, 2020 (9 pages). [cited by applicant]