IP Library › Granted Patent US 12,387,094
Granted Patent B2
US 12,387,094 · App. 17/432,618 · Granted Aug 12, 2025

Photonic tensor accelerators for artificial neural networks

Inventors: Guifang Li (Orlando, FL); Shuo Pang (Orlando, FL); He Wen (Orlando, FL)
Assignee: University of Central Florida Research Foundation, Inc.
G06N3/0675G06N3/048H04B10/54H04B10/58
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,387,094
App. No.
17/432,618
Granted
Aug 12, 2025
Kind
B2
Abstract

Photonic units for vector-vector multiplication, matrix-vector multiplication, matrix-matrix multiplication, batch matrix-matrix multiplication, and tensor-tensor multiplication are described. Multiplications are through coherent mixing and square-law detection. There are many dimensions—wavelength, vector mode, quadrature, and three dimensions of space—that can be used to construct photonic accelerators. The encoded input vector or input matrix is fanned out into a desired number of copies and mixed with the corresponding encoded local oscillators containing the weight vectors comprising the weight matrix. Any subset of two (three) dimensions can be used to construct photonic accelerators for matrix-vector (matrix-matrix) multiplications. Multiple dimensions can be combined into a hyperdimension to increase the scalability. Each dimension, each non-overlapping subset of a dimension, or each non-overlapping subset of a hyperdimension, can be used independently to construct a photonic tensor accelerator (PTA) for batch matrix multiplication, or tensor multiplication operations.

Claims (44)

1. A photonic unit for vector multiplication comprising:

a first optical multiplexer that receives and multiplexes a first set of optical signals to produce a first multiplexed optical signal representing a first vector, wherein each element in the first vector is encoded on a different optical signal of the first set of optical signals, each optical signal of the first set of optical signals have a different degree of freedom (DOF)/dimension of light to the other optical signals of the first set of optical signals and is non-temporal during one multiplication cycle;

a second optical multiplexer that receives and multiplexes a second set of optical signals, to produce a second multiplexed optical signal, representing a second vector, each element in the second vector is encoded on a different optical signal of the second set, wherein each optical signal of the second set of optical signals has a different degree of freedom (DOF)/dimension to the other optical signals of the second set, and that is identical to one of the optical signals of the first set of optical signals and is non-temporal during a vector multiplication cycle; and

a beam combiner that receives and combines the first multiplexed optical signal from the first optical multiplexer and second multiplexed optical signal from the second optical multiplexer to produce an interference between the first multiplexed optical signal and second multiplexed optical signal containing multiplication results of the first vector and the second vector in a total interference intensity.

2. The photonic unit of claim 1 , wherein each optical signal of the first set of optical signals have the different degree of freedom (DOF)/dimension of light used for encoding thereof is one of a wavelength, a spatial mode, a polarization, a quadrature, a component of a wave vector, or a combination thereof.

3. The photonic unit of claim 1 , wherein each optical signal of the first set of optical signals have the different degree of freedom (DOF)/dimension of light used for encoding is a hyperdimension consisting of combinations of two or more degree of freedoms (DOFs)/dimensions of light.

4. The photonic unit of claim 2 , wherein the spatial mode is one of:

Hermite-Gaussian modes,

Laguerre-Gaussian modes,

discrete spatial samples forming spatially orthonormal basis, or

a combination thereof.

5. The photonic unit of claim 2 , wherein an interference signal enters a nonlinear optical element.

6. The photonic unit of claim 2 , wherein the total interference intensity is converted into an electrical signal.

7. The photonic unit of claim 6 , wherein the electrical signal enters a nonlinear electrical element.

8. A photonic unit for N by M matrix-with M by 1 vector multiplication comprising:

a first optical multiplexer that receives and multiplexes a first set of optical signals to produce a first multiplexed optical signal representing a M by 1 vector with M elements, wherein each element in the M by 1 vector is encoded on a different optical signal of the first set of optical signals, each optical signal of the first set of optical signals having a different orthogonal degree of freedom (DOF)/dimension of light to the other optical signals of the first set of optical signals and is non-temporal during one matrix-vector multiplication cycle, and wherein M is a positive integer greater than or equal to 1;

an optical duplicator for reproducing the first multiplexed optical signal representing the M by 1 vector to provide N copies, wherein N is a positive integer greater than 1;

a set of N additional optical multiplexers each receiving and multiplexing a different set of M optical signals from a group of N sets of M optical signals to produce N multiplexed optical signals, wherein each of set of M optical signals represents a different independent row of the M by N matrix, and wherein each element in a row of the M by N matrix is encoded on a different optical signal of its respective set of M optical signals, and wherein each o I signal of each set of M optical signals has a different degree of freedom (DOF)/dimension of light and that is identical to the degree of freedom (DOF)/dimension of light of one of the optical signals of the first set of optical signals and is no-temporal during a matrix-vector multiplication cycle; and

at least one beam combiner that receives the N copies of the first multiplexed optical signal and the N second multiplexed optical signals and combines them to produce N occurrences of interferences between the first optical signal and each of the N second optical signals containing multiplication results of the M by N matrix and the M by 1 vector in N total interference intensities.

9. The photonic unit of claim 8 , wherein at least one of the first set of optical signals and the second set of optical signals that uses different orthogonal degrees of freedom (DOF)/dimension of is one of a wavelength, a spatial mode, a polarization, a quadrature, a component of a wave vector, or combination thereof.

10. The photonic unit of claim 8 , wherein the at least one of the first set of optical signals and the second set of optical signals that uses different orthogonal degree of freedom (DOF)/dimension of light is a hyperdimension consisting of combinations of two or more degree of freedoms (DOFs)/dimensions of light.

11. The photonic unit of claim 8 , wherein the first set of optical signals and the second set of optical signals that uses different orthogonal degree of freedom (DOF)/dimension of light are non-overlapping subsets of a dimension or a hyperdimension of light.

12. The photonic unit of claim 9 , wherein the spatial mode is one of:

Hermite-Gaussian modes,

Laguerre-Gaussian modes,

discrete spatial samples forming spatially orthonormal basis, or

a combination thereof.

13. The photonic unit of claim 8 , wherein at least one interference signal enters a nonlinear optical element.

14. The photonic unit of claim 8 , wherein at least one total interference intensity is converted into an electrical signal.

15. The photonic unit of claim 14 , wherein the electrical signal enters a nonlinear electrical element.

16. A photonic unit for a N by M matrix-with a M by W matrix multiplication comprising:

a set of N optical multiplexers each receiving and multiplexing a different set of M optical signals from a group of N sets of M optical signals to produce N multiplexed optical signals, wherein each of set of M optical signals represents a different independent row with M elements of a N by M matrix, and wherein each element in each independent row of the N by M matrix is encoded on a different optical signal of its respective set of M optical signals, and wherein each of optical signal of each set of M optical signals has a different degree of freedom (DOF)/dimension of light and that is identical to the degree of freedom (DOR)/dimension of light of optical signals of the set of N multiplexed optical signals and is non-temporal during one matrix-matrix multiplication cycle, and wherein M and N are each a positive integer greater than or equal to 1;

a first optical duplicator for reproducing each of the N multiplexed optical signals to provide W copies, wherein W is a positive integer greater than 1;

a set of W optical multiplexers each receiving and multiplexing a different set of M optical signals from a group of W sets of M optical signals to produce W multiplexed optical signals, wherein each of the set of W optical signals represents a different independent column with M elements of the M by W matrix, and wherein each element in each independent column of the M by W matrix is encoded on a different optical signal of its respective set of W optical signals, and herein each optical signal of each set of W optical signals has a different degree of freedom (DOF)/dimension of light and that is identical to degree of freedom (DOF)/dimension of light of one of the optical signals of the set of W multiplexed optical signals and is non-temporal during matrix-matrix multiplication cycle;

a second optical duplicator for reproducing each of the W multiplexed optical signals to provide N copies wherein W is the positive integer greater than 1; and

at least one beam combiner that receives two sets of N by W multiplexed optical signals representing appropriately duplicated rows or columns of each of the N by M matrix and the M by W matrix so as to combine them to produce N by W occurrences of interferences between each of row of the N by M matrix and the column of the M by W matrix containing multiplication results in N by W total interference intensities.

17. The photonic unit of claim 16 , wherein the at least one of the first, the second, and the third orthogonal degree of freedom (DOF)/dimension of light used for encoding, or duplication is one of a wavelength, a spatial mode, a polarization, a quadrature, a component of a wave vector, or combination thereof.

18. The photonic unit of claim 16 , wherein the at least one of the first, the second, and the third orthogonal degree of freedom (DOF)/dimension used for encoding, or duplication is a hyperdimension consisting of combinations of two or more degree of freedoms (DOFs)/dimensions of light.

19. The photonic unit of claim 16 , wherein at least two of the first, the second and the third orthogonal degree of freedom (DOF)/dimension of light used for encoding, or duplication are non-overlapping subsets of a dimension or a hyperdimension of light.

20. The photonic unit of claim 17 , wherein the spatial mode is one of:

Hermite-Gaussian modes,

Laguerre-Gaussian modes,

discrete spatial samples forming spatially orthonormal basis, or

combination thereof.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 9, 2021
From: LI, GUIFANG; PANG, SHUO; WEN, HE
To: UNIVERSITY OF CENTRAL FLORIDA RESEARCH FOUNDATION, INC.
Reel/Frame 057425/0880 →
Continuity (2)
Provisional Application 62842771 · May 3, 2019
Related Publication 20220164642A1 · May 26, 2022
References Cited (83)
US 7515828B2 · Gumaste et al. · 2009 [cited by applicant]
US 8027587B1 · Watts · 2011 [cited by applicant]
US 9354039B2 · Mower et al. · 2016 [cited by applicant]
US 9791258B2 · Mower et al. · 2017 [cited by applicant]
US 10359272B2 · Mower et al. · 2019 [cited by applicant]
US 10619993B2 · Mower et al. · 2020 [cited by applicant]
US 20110194172A1 · Esteban Martin · 2011 [cited by examiner]
US 20140016168A1 · Marandi · 2014 [cited by examiner]
US 20140199065A1 · Bratkovski · 2014 [cited by examiner]
US 20170366270A1 · Ashrafi · 2017 [cited by examiner]
US 20180069631A1 · Ashrafi · 2018 [cited by examiner]
US 20180107237A1 · Andregg et al. · 2018 [cited by applicant]
US 20190331912A1 · Tait · 2019 [cited by examiner]
US 20190354894A1 · Lazovich · 2019 [cited by examiner]
US 20190370644A1 · Kenney · 2019 [cited by examiner]
US 20200085285A1 · Yamada · 2020 [cited by examiner]
US 20220094443A1 · Bunandar · 2022 [cited by examiner]
US 20220366308A1 · Lazovich · 2022 [cited by examiner]
WO WO2016028363 · 2016 [cited by applicant]
Hamerly et al. Large-Scale Optical Neural Networks Based on Photoelectric Multiplication, Nov. 12, 2018, Physical Review, All (Year: 2018). [cited by examiner]
Yang et al. On-chip CMOS-compatible optical signal processor; Jun. 4, 2012, Optical Society of America, vol. 20, No. 12 (Year: 2012). [cited by examiner]
Bienvenu I. Ndagano; Encoding information into spatial modes of light; May 3, 2016; University of the Witwatersrand, pp. 1-90 (Year : 2016). [cited by examiner]
George et al.; Electrooptic Nonlinear Activation Functions for Vector Matrix Multiplications in Optical Neural Networks; 2018; pp. 1-2 (Year: 2018). [cited by examiner]
George et al. (Year: 2018). [cited by examiner]
Bienvenu Ndagano (Year: 2016). [cited by examiner]
Hamerly et al (Year: 2018). [cited by examiner]
Yang et al. (Year: 2012). [cited by examiner]
Martin Abadi, et al., TensorFlow: A system for large-scale machine learning, Google Brain, May 31, 2016, 18 pages. [cited by applicant]
T.A. Birks, et al., The Photonic Lantern, Advances in Optics and Photonoics 7, 107-167 (2015) dol: 10. 1364/AOP.7.000107, Received Dec. 16, 2014, 61 pages. [cited by applicant]
Z.I. Borevich and S.L. Krupetskii, Subgroups of the Unitary Group that Contain the Group of Diagonal Matrices, Plenum Publishing Corporation, 1981, 9 pages. [cited by applicant]
D.J. Brady, et al., Multiscale gigapixel photography, Nature, vol. 486, Jun. 21, 2012, 4 pages, Macmillan Publishers Limited 2012. [cited by applicant]
H. John Caulfield and Shlmi Dolev, Why future supercomputing requires optics, Magazine, Nature Photonics, vol. 4, May 2010, Commentary section, pp. 261-263. [cited by applicant]
Antoine Dejonckheere, et al., All-optical reservoir computer based on saturation of absorption, Optical Society of America, vol. 22, No. 9, Published on Apr. 29, 2014, pp. 10868-10881. [cited by applicant]
Francois Deport, et al., All-optical reservoir computing, 2012 Optical Society of America, Published Sep. 20, 2012, vol. 20, No. 20, pp. 22783-22795. [cited by applicant]
Nicholas K. Fontaine, et al., Scalable mode sorter supporting 210 Hermite-Gaussian modes, Optical Society of America, Preliminary Paper, 3 Pages. [cited by applicant]
Liang Gao, Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature vol. 516, Dec. 4, 2014, pp. 77-85. [cited by applicant]
Jonathan K. George, et al., Neuromorphic photonics with electro-absorption modulators, Optics Express 5181, vol. 27 No. 4, Dated Feb. 18, 2019, 11 Pages. [cited by applicant]
Joseph W. Goodman, Operations Achievable with Coherent Optical Information Processing Systems, Proceedings of the IEEE, vol. 65 No. 1, Jan. 1977, 10 Pages. [cited by applicant]
Ryan Hamerly, et al., Large-Scale Optical Neural Networks based on Photoelectric Multiplication, Research Laboratory of Electronics, MIT, Cambridge, MA, Dec. 20, 2018, 18 Pages. [cited by applicant]
Geoffrey Hinton, et al., Deep Neural Networks for Acoustic Modeling in Speech Recognition, IEEE Signal Processing Magazine, Nov. 2012, pp. 82-97. [cited by applicant]
Gin Huang, et al., Structured Directional Coupler Pair for Multiplexing of Degenerate Modes, OFC/NFOEC Technical Digest 2013, Published in the US, 3 Pages. [cited by applicant]
Bin Huang, et al., All-fiber mode-group-selective photonic lantern using graded-index multimode fibers., Optics Express 224, Jan. 12, 2015, vol. 23, No. 1, 11 Pages., Journal. [cited by applicant]
Bin Huang, et al., Triple-clad photonic lanterns for mode scaling., Optics Express 13391, Research Article, vol. 26, No. 10, May 14, 2018, 7 Pages. [cited by applicant]
M.N. Islam, Ultrafast all-optical logic gates based on soliton trapping in fibers, Optical Society of America, Optics Letters, vol. 14, No. 22, Nov. 15, 1989, 3 Pages. [cited by applicant]
Bade et al., “Fabrication and Characterization of a Mode-selective 45-Mode Spatial Multiplexer based on Multi-Plane Light Conversion”, Optical Society of America, 2018. [cited by applicant]
C Gao et al., “Non-circularly-symmetric Mode-group Demultiplexer Based on Fused-type FMF Coupler for MGM Transmission”, Optical Society of America, 2018. [cited by applicant]
Brunner et al., “All-Optical Neuromorphic Computing in Optical Networks of Semiconductor Lasers”, 2014. [cited by applicant]
Deng et al., “Controlled Propagation of Spiking Dynamics in Vertical-Cavity Surface-Emitting Lasers: Towards Neuromorphic Photonic Networks”, IEEE Journal of Selected Topics in Quantum Electronics, vol. 23, No. 6, 2017. [cited by applicant]
Jia et al., “Caffe: Convolutional Architecture for Fast Feature Embedding”, ACM, 2014. [cited by applicant]
Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit”, ISCA, Jun. 2017. [cited by applicant]
Jain et al., “Optical transistor”, Applied Physical Letters, 28, 719, American Institute of Physics, Aug. 28, 2008. [cited by applicant]
Katumba et al., “Low-Loss Photonic Reservoir Computing with Multimode Photonic Integrated Circuits”, Scientific Reports, Feb. 8, 2018. [cited by applicant]
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks”, 2012. [cited by applicant]
Abroille et al., “Efficient and mode selective spatial mode multiplexer based on multi-plane light conversion”, Optical Society of America, 2014. [cited by applicant]
Larger et al., “High-Speed Photonic Reservoir Computing Using a Time-Delay-Based Architecture: Million Words per Second Classification”, American Physical Society, 011015, 2017. [cited by applicant]
Ferreira De Lima et al., “Progress in neuromorphic photonics”, Nanophotonics, pp. 577-599, 2017. [cited by applicant]
X. Lin et al., “All-optical machine learning using diffractive deep neural networks,” Science (1004-1008), 2018. [cited by applicant]
S. A. Manavski, “CUDA compatible GPU as an efficient hardware accelerator for AES cryptography,” in ICSPC 2007 Proceedings—2007 IEEE International Conference on Signal Processing and Communications, 2007. [cited by applicant]
M. Mathieu, M. Henaff, and Y. LeCun, “Fast Training of Convolutional Networks through FFTs,” pp. 1-9, 2013. [cited by applicant]
P. Merolla, R. Appuswamy, J. Arthur, S. K. Esser, and D. Modha, “Deep neural networks are robust to weight pinarization and other non-linear distortions,” Jun. 2016. [cited by applicant]
D. A. B. Miller, “Are optical transistors the logical next step?,” Nat. Photonics, vol. 4, No. 1, pp. 3-5, Jan. 2010. [cited by applicant]
J.-F. Morizur et al., “Programmable unitary spatial mode manipulation,” J. Opt. Soc. Am. A, vol. 27, No. 11, p. 2524, Nov. 2010. [cited by applicant]
Y. Paquot et al., “Optoelectronic reservoir computing,” Sci. Rep., 2012. [cited by applicant]
H. T. Peng, M. A. Nahmias, T. F. De Lima, A. N. Tait, B. J. Shastri, and P. R. Prucnal, “Neuromorphic Photonic Integrated Circuits,” IEEE J. Sel. Top. Quantum Electron., 2018. [cited by applicant]
L. Larger et al., “Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing,” Opt. Express, vol. 20, No. 3, p. 3241, 2012. [cited by applicant]
D. Psaltis, D. Brady, X. G. Gu, and S. Lin, “Holography in artificial neural networks.,” Nature, vol. 343, No. 6256, pp. 325-330, 1990. [cited by applicant]
G. Quintana-Orti, F. D. Igual, E. S. Quintana-Orti, and R. A. van de Geijn, “Solving dense linear systems on platforms with multiple hardware accelerators,” ACM SIGPLAN Not., 2009. [cited by applicant]
M. Reck, A. Zeilinger, H. J. Bernstein, and P. Bertani, “Experimental realization of any discrete unitary operator,” Phys. Rev. Lett., 1994. [cited by applicant]
R. F. Rutz, “Transistor-Like Device Using Optical Coupling Between Diffused P-N Junctions in GaAs,” Proc. IEEE, 1963. [cited by applicant]
Y. Shen et al., “Deep learning with coherent nanophotonic circuits,” Nat. Photonics, vol. 11, No. 7, pp. 441-446, Jul. 2017. [cited by applicant]
D. Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, 2016. [cited by applicant]
K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” pp. 1-14, 2014. [cited by applicant]
A. N. Tait et al., “Neuromorphic photonic networks using silicon photonic weight banks,” Sci. Rep., 2017. [cited by applicant]
H. Takahashi, T. Saida, Y. Sakamaki, and T. Hashimoto, “Wavefront matching method: A new approach for need- priented waveguide design,” in Conference Proceedings—Lasers and Electro-Optics Society Annual Meeting-LEOS, 20… [cited by applicant]
H. F. Taylor, “Guided wave electrooptic devices for logic and computation,” Appl. Opt., vol. 17, No. 10, p. 1493, May 1978. [cited by applicant]
T. Umezawa et al., “10-GHz 32-pixel 2-D photodetector array for advanced optical fiber communications,” 2017. [cited by applicant]
K. Vandoorne et al., “Toward optical signal processing using Photonic Reservoir Computing,” Opt. Express, vol. 16, No. 15, p. 11182, Jul. 2008. [cited by applicant]
K. Wagner and D. Psaltis, “Multilayer optical learning networks,” Appl. Opt., vol. 26, No. 23, p. 5061, Dec. 1987. [cited by applicant]
Y. Wang, N. Zhao, Z. Yang, Z. Zhang, B. Huang, and G. Li, “Few-mode SDM receivers exploiting parallelism of free space,” IEEE Photonics Journal, 2019. [cited by applicant]
Written Opinion of the International Searching Authorities for PCT Application No. PCT/US2019/56239, dated Jan. 14, 2020. [cited by applicant]
Bienvenu I. Ndagano, “Encoding information into spatial modes of light”, School of Physics, University of Witwatersand, disseration submitted for degree of Master of Science, May 2, 2016, pp. 1-74. [cited by applicant]
In Yang, et al., “On-chip CMOS-compatible optical signal processor”, Optics Express, Jun. 34, 2012, vol. 20, No. 12. pp. 1-6. [cited by applicant]
Supplemental International Search Report, dated Jun. 7, 2022, Reference P657.EP. [cited by applicant]