Devices and methods employing optical-based machine learning using diffractive deep neural networks
An all-optical Diffractive Deep Neural Network (D 2 NN) architecture learns to implement various functions or tasks after deep learning-based design of the passive diffractive or reflective substrate layers that work collectively to perform the desired function or task. This architecture was successfully confirmed experimentally by creating 3D-printed D 2 NNs that learned to implement handwritten classifications and lens function at the terahertz spectrum. This all-optical deep learning framework can perform, at the speed of light, various complex functions and tasks that computer-based neural networks can implement, and will find applications in all-optical image analysis, feature detection and object classification, also enabling new camera designs and optical components that can learn to perform unique tasks using D 2 NNs. In alternative embodiments, the all-optical D 2 NN is used as a front-end in conjunction with a trained, digital neural network back-end.
1. A hybrid optical and electronic neural network-based system comprising:
an all-optical front-end comprising a plurality of optically transmissive substrate layers arranged in an optical path, each of the plurality of optically transmissive substrate layers comprising a plurality of physical features formed on or within the plurality of optically transmissive substrate layers having different complex-valued transmission coefficients as a function of lateral coordinates across each substrate layer, wherein the plurality of optically transmissive substrate layers and the plurality of physical features collectively define a trained mapping function between an input optical image or input optical signal to the plurality of optically transmissive substrate layers and an output optical image or output optical signal created by optical diffraction through the plurality of optically transmissive substrate layers;
one or more optical sensors configured to capture the output optical image or output optical signal resulting from the plurality of optically transmissive substrate layers; and
a trained, digital neural network configured to receive as an input the output optical image or output optical signal resulting from the plurality of optically transmissive substrate layers and output a final output image or final output signal.
2. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of optically transmissive substrate layers are separated from one another by a gap.
3. The hybrid optical and electronic neural network-based system of claim 1 , wherein the input optical image or input optical signal is generated by another separate optical imaging system, at least one lens, or a projection of optical images or optical signals of interest onto an input plane of the all-optical front-end of the hybrid system.
4. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of optically transmissive substrate layers comprise a monolithic 3D structure.
5. The hybrid optical and electronic neural network-based system of claim 1 , wherein each optically transmissive substrate layer defines a planar or non-planar surface.
6. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of optically transmissive substrate layers are mounted or held within a holder.
7. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of physical features of the plurality of optically transmissive substrate layers are formed by additive manufacturing.
8. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of physical features of the plurality of optically transmissive substrate layers are lithographically formed.
9. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of physical features comprises an array of neurons formed on or in the optically transmissive substrate layers.
10. The hybrid optical and electronic neural network-based system of claim 1 , wherein the plurality of physical features comprises an array of neurons formed from an optically non-linear material.
11. The hybrid optical and electronic neural network-based system of claim 1 , wherein the trained mapping function comprises an imaging function acting on the phase and/or amplitude channels of the input optical image or input optical signal.
12. The hybrid optical and electronic neural network-based system of claim 1 , wherein the trained mapping function is generated using deep learning software trained using a set of training images or data.
13. The hybrid optical and electronic neural network-based system of claim 1 , wherein the trained mapping function comprises the physical locations and the transmission coefficients including both phase and amplitude of an array of neurons formed by the plurality of physical features of the plurality of optically transmissive substrate layers.
14. A hybrid optical and electronic neural network-based system comprising:
an all-optical front-end comprising a plurality of optically reflective substrate layers arranged along an optical path, each of the plurality of optically reflective substrate layers comprising a plurality of physical features, wherein the plurality of optically reflective substrate layers and the plurality of physical features thereon collectively define a trained mapping function between an input optical image or input optical signal to the plurality of optically reflective substrate layers and an output optical image or output optical signal from the plurality of optically reflective substrate layers;
one or more optical sensors configured to capture the output optical image or output optical signal from the plurality of optically reflective substrate layers; and
a trained digital neural network configured to receive as an input the output optical image or output optical signal resulting from the plurality of optically reflective substrate layers and output a final output image or final output signal.
15. The hybrid optical and electronic neural network-based system of claim 14 , wherein each optically reflective substrate layer defines a planar or non-planar surface.
16. The hybrid optical and electronic neural network-based system of claim 14 , wherein the input optical image or input optical signal is generated by another separate optical imaging system, at least one lens, or a projection of optical images or optical signals of interest onto an input plane of the all-optical front-end of the hybrid system.
17. The hybrid optical and electronic neural network-based system of claim 14 , wherein the plurality of physical features of the plurality of optically reflective substrate layers are formed by additive manufacturing.
18. The hybrid optical and electronic neural network-based system of claim 14 , wherein the plurality of physical features of the plurality of optically reflective substrate layers are lithographically formed.
19. The hybrid optical and electronic neural network-based system of claim 14 , wherein the plurality of physical features comprise an array of neurons formed on or in the optically reflective substrate layers.
20. The hybrid optical and electronic neural network-based system of claim 14 , wherein the plurality of physical features comprise an array of neurons formed from an optically non-linear material.
21. The hybrid optical and electronic neural network-based system of claim 14 , wherein the trained mapping function is generated using deep neural network software trained using a set of training images or data.
22. The hybrid optical and electronic neural network-based system of claim 21 , wherein the trained mapping function comprises the physical locations and the transmission coefficients including both phase and amplitude of an array of neurons formed by the plurality of physical features of the plurality of optically reflective substrate layers.