Meta-optic accelerators for object classifiers
A system for identifying objects in images is provided. The system may include an optical front end and a digital back end. The optical front end includes a metalens that duplicates a received image into multiple images, and a metasurface that receives the duplicate images and outputs a feature map based on the received images. The feature map may be equivalent to the computationally expensive convolution operations previously performed by a neural network. The feature map is provided to the digital back end, which uses a neural network to classify the object. Because the feature map included the convolution operations, the digital back end can classify the object more quickly and using fewer computing resources than previous systems.
1 . An object classification system comprising:
an optical front end adapted to;
receive an image of an object; and
output a feature map, or maps, based on the image; and
a digital back end adapted to:
receive the feature map from the optical front end;
process the feature map using a neural network; and
output a classification for the object, wherein the optical front end comprises:
a lens adapted to duplicate the received image into a plurality of duplicate images; and
a kernel layer adapted to receive the plurality of duplicate images and to output the feature map based on the plurality of duplicate images, wherein the kernel layer is a metasurface that encodes polarization and amplitude information for convolution with the plurality of duplicate images, wherein the metasurface comprises a plurality of nanopillars, and the metasurface is trained using a set of training data and wherein a rotation angle of some or all of the plurality of nanopillars are set during the training.
2 . The system of claim 1 , wherein the plurality of duplicate images comprises multiple images.
3 . The system of claim 1 , wherein the lens is a multi-channel metalens.
4 . The system of claim 1 , wherein the kernel layer replaces a multifunctional kernel layer of the neural network.
5 . The system of claim 1 , further comprising:
receiving the set of training data; and
training the metasurface and the neural network using the set of training data.
6 . The system of claim 1 , wherein the neural network comprises a shallow or deep digital neural network.
7 . The system of claim 1 , wherein the classification for the object comprises a histogram.
8 . The system of claim 1 , wherein the system comprises a digital camera.
9 . The object classification system of claim 1 , wherein the lens comprises a single multi-channel metalens.
10 . The object classification system of claim 1 , wherein the lens comprises a plurality of silicon nanopillars.
11 . The object classification system of claim 1 , wherein the lens comprises a super-cell comprising a plurality of resonators (i), wherein a phase delay of each resonator is given by the equation:
ϕ
i
=
2
π
λ
(
f
-
f
2
+
(
x
-
a
i
)
2
+
(
y
-
b
i
)
2
)
where f is a focal length of the lens, λ is the working wavelength of the lens, x and y are spatial positions on the lens, and a and b correspond to a displacement of a unique focal spot corresponding to each resonator (i).
12 . A method for classifying an object in an image comprising:
receiving an image of an object by an optical front end;
outputting a feature map, or maps, based on the image by the optical front end;
receiving the feature map from the optical front end by a digital back end;
processing the feature map using a neural network by the digital back end; and
outputting a classification for the object by the digital back end, wherein the optical front end comprises:
a lens adapted to duplicate the received image into a plurality of duplicate images; and
a kernel layer adapted to receive the plurality of duplicate images and to output the feature map based on the plurality of duplicate images, wherein the kernel layer is a metasurface that encodes polarization and amplitude information for convolution with the plurality of duplicate images, wherein the metasurface comprises a plurality of nanopillars, and the metasurface is trained using a set of training data and wherein a rotation angle of some or all of the plurality of nanopillars are set during the training.
13 . The method of claim 12 , wherein the plurality of duplicate images comprises nine or more duplicate images.
14 . The method of claim 12 , wherein the lens is a multi-channel metalens.
15 . The method of claim 12 , wherein the metasurface replaces a multifunctional kernel layer of the neural network.
16 . The method of claim 12 , further comprising:
receiving the set of training data; and
training the metasurface and the neural network using the set of training data.
17 . The method of claim 12 , wherein the neural network comprises a shallow or deep digital neural network.
18 . The method of claim 12 , wherein the classification for the object comprises a histogram.
19 . A digital camera comprising:
an optical front end adapted to;
receive an image of an object; and
output a feature map, or maps, based on the image; and
a digital back end adapted to:
receive the feature map from the optical front end;
process the feature map using a neural network; and
output a classification for the object, wherein the optical front end comprises a lens adapted to duplicate the received image into a plurality of duplicate images; and
a metasurface adapted to receive the plurality of duplicate images and to output the feature map based on the plurality of duplicate images, and wherein the metasurface replaces a multifunctional kernel layer of the neural network, and wherein the metasurface encodes polarization and amplitude information for convolution with the plurality of duplicate images, wherein the metasurface comprises a plurality of nanopillars, and the metasurface is trained using a set of training data and wherein a rotation angle of some or all of the plurality of nanopillars are set during the training.