Convolutional layers for neural networks using programmable nanophotonics
Aspects of the present application relate to techniques for computing convolutions and cross-correlations of input matrices. A first technique is based on the transformation of convolution operations into a matrix-vector product. A second technique is based on two-dimensional matrix multiplication. A third technique is based on the convolution theorem, which states that convolutions correspond to multiplications in a transform space. Embodiments include methods for computing convolutions of a filter matrix and an input data matrix; apparatuses for computing convolutions of a filter matrix and an input data matrix; and a non-transitory computer readable medium programmed with instructions that, when executed by a processor perform a method for computing convolutions of a filter matrix and an input data matrix.
1. A method performed by a photonic processing system, the method comprising:
pre-processing at least one input signal and/or at least one filter signal by expanding and/or flattening the at least one input signal and/or the at least one filter signal;
computing, using the photonic processing system, at least one of a convolution and a cross-correlation on the at least one input signal and the at least one filter signal by performing a matrix multiplication operation, wherein:
the at least one input signal and at least one filter signal are at least one dimensional and comprise at least one data channel; and
the at least one convolution and cross-correlation produce an at least one output signal that is at least one dimensional and comprises at least one data channel; and
post-processing the at least one output signal by rotating vector rows of a matrix forming the at least one output signal, the rotating comprising shifting matrix element positions within the vector rows.
2. The method of claim 1 , wherein the method further comprises structuring the matrix multiplication operation using at least one of a selection of an image to column (im2col) algorithm, a kernel to row (kn2row) algorithm, and a memory-efficient convolution (MEC) algorithm.
3. A processing system, comprising:
at least one photonic processor; and
at least one non-transitory computer-readable medium comprising instructions, which, when executed by the at least one photonic processor, cause the at least one photonic processor to perform a method of:
pre-processing at least one input signal and/or at least one filter signal by expanding and/or flattening the at least one input signal and/or the at least one filter signal;
computing at least one of a convolution and a cross-correlation on the at least one input signal and the at least one filter signal by performing a matrix multiplication operation, wherein:
the at least one input signal and at least one filter signal are at least one dimensional and comprise at least one data channel; and
the at least one convolution and cross-correlation produce an at least one output signal that is at least one dimensional and comprises at least one data channel; and
post-processing the at least one output signal by rotating vector rows of a matrix forming the at least one output signal, the rotating comprising shifting matrix element positions within the vector rows.
4. The processing system of claim 3 , wherein the method further comprises structuring the matrix multiplication operation using at least one of a selection of an image to column (im2col) algorithm, a kernel to row (kn2row) algorithm, and a memory-efficient convolution (MEC) algorithm.
5. At least one non-transitory computer-readable medium comprising instructions, which, when executed by an at least one photonic processor, cause the at least one photonic processor to perform a method of:
pre-processing at least one input signal and/or at least one filter signal by expanding and/or flattening the at least one input signal and/or the at least one filter signal;
computing at least one of a convolution and a cross-correlation on the at least one input signal and the at least one filter signal by performing a matrix multiplication operation, wherein:
the at least one input signal and at least one filter signal are at least one dimensional and comprise at least one data channel; and
the at least one convolution and cross-correlation produce an at least one output signal that is at least one dimensional and comprises at least one data channel; and
post-processing the at least one output signal by rotating and/or shifting vector rows of a matrix forming the at least one output signal, the rotating comprising shifting matrix element positions within the vector rows.
6. The at least one non-transitory computer-readable medium of claim 5 , wherein the method further comprises structuring the matrix multiplication operation using at least one of a selection of an image to column (im2col) algorithm, a kernel to row (kn2row) algorithm, and a memory-efficient convolution (MEC) algorithm.
7. A method performed by a photonic processing system, the method comprising:
pre-processing at least one input signal and/or at least one filter signal by expanding and/or flattening the at least one input signal and/or the at least one filter signal;
computing, using the photonic processing system, at least one of a convolution and a cross-correlation on the at least one input signal and the at least one filter signal, wherein:
the at least one input signal and at least one filter signal comprise at least one data channel;
computing at least one of a convolution and a cross-correlation comprises applying a convolution theorem and a filtering transform operation; and
the at least one convolution and cross-correlation produces an at least one output signal that comprises at least one data channel; and
post-processing the at least one output signal by rotating vector rows of a matrix forming the at least one output signal, the rotating comprising shifting matrix element positions within the vector rows.
8. The method of claim 7 , wherein the at least one input signal and at least one filter signal may be at least one of complex-valued and real-valued.
9. The method of claim 7 , wherein the filtering transform operation is at least one of a selection of a discrete Fourier transform, a discrete sine transform, a discrete cosine transform, an undecimated wavelet transform, a discrete Hartley transform, a Hankel transform, a Walsh-Hadamard transform, and Winograd's minimal filtering algorithm.
10. The method of claim 7 , wherein the method requires multiple passes through the photonic processing system, and wherein the photonic processing system represents real-valued operands.
11. The method of claim 7 , further comprising:
computing at least one transform operation of a selection of the at least one input signal and at least one filter signal on a photonic processor of the photonic processing system; and
computing at least one transform-domain product on the photonic processor, wherein:
the at least one input signal and at least one filter signal comprise multiple channels; and
corresponding matrix multiplications in a transform domain are executed on the photonic processor.
12. A processing system, comprising:
at least one photonic processor; and
at least one non-transitory computer-readable medium comprising instructions, which, when executed by the at least one photonic processor, cause the at least one photonic processor to perform a method of:
pre-processing at least one input signal and/or at least one filter signal by expanding and/or flattening the at least one input signal and/or the at least one filter signal;
computing at least one of a convolution and a cross-correlation on the at least one input signal and the at least one filter signal, wherein:
the at least one input signal and at least one filter signal comprise at least one data channel;
computing at least one of a convolution and a cross-correlation comprises applying a convolution theorem and a filtering transform operation; and
the at least one convolution and cross-correlation produces an at least one output signal that comprises at least one data channel; and
post-processing the at least one output signal by rotating vector rows of a matrix forming the at least one output signal, the rotating comprising shifting matrix element positions within the vector rows.
13. The processing system of claim 12 , wherein the at least one input signal and at least one filter signal may be at least one of complex-valued and real-valued.
14. The processing system of claim 12 , wherein the filtering transform operation is at least one of a selection of a discrete Fourier transform, a discrete sine transform, a discrete cosine transform, an undecimated wavelet transform, a discrete Hartley transform, a Hankel transform, a Walsh-Hadamard transform, and Winograd's minimal filtering algorithm.
15. The processing system of claim 12 , wherein the method requires multiple passes through the at least one photonic processor, and wherein the at least one photonic processor represents real-valued operands.
16. The processing system of claim 12 , wherein the method further comprises:
computing at least one transform operation of a selection of the at least one input signal and at least one filter signal using the at least one photonic processor; and
computing at least one transform-domain product using the at least one photonic processor, wherein:
the at least one input signal and at least one filter signal comprise multiple channels; and
corresponding matrix multiplications in a transform domain are executed on the at least one photonic processor.
17. At least one non-transitory computer-readable medium comprising instructions, which, when executed by an at least one photonic processor, cause the at least one photonic processor to perform a method of:
pre-processing at least one input signal and/or at least one filter signal by expanding and/or flattening the at least one input signal and/or the at least one filter signal;
computing, using the at least one photonic processor, at least one of a convolution and a cross-correlation on the at least one input signal and the at least one filter signal, wherein:
the at least one input signal and at least one filter signal comprise at least one data channel;
computing at least one of a convolution and a cross-correlation comprises applying a convolution theorem and a filtering transform operation; and
the at least one convolution and cross-correlation produces an at least one output signal that comprises at least one data channel; and
post-processing the at least one output signal by rotating vector rows of a matrix forming the at least one output signal, the rotating comprising shifting matrix element positions within the vector rows.
18. The at least one non-transitory computer-readable medium of claim 17 , wherein the at least one input signal and at least one filter signal may be at least one of complex-valued and real-valued.
19. The at least one non-transitory computer-readable medium of claim 17 , wherein filtering transform operation is at least one of a selection of a discrete Fourier transform, a discrete sine transform, a discrete cosine transform, an undecimated wavelet transform, a discrete Hartley transform, a Hankel transform, a Walsh-Hadamard transform, and Winograd's minimal filtering algorithm.
20. The at least one non-transitory computer-readable medium of claim 17 , wherein the method requires multiple passes through the at least one photonic processor, and wherein the at least one photonic processor represents real-valued operands.
21. The at least one non-transitory computer-readable medium of claim 17 , the method further comprising:
computing at least one transform operation of a selection of the at least one input signal and at least one filter signal using the at least one photonic processor; and
computing at least one transform-domain product using the at least one photonic processor, wherein:
the at least one input signal and at least one filter signal comprise multiple channels; and
corresponding matrix multiplications in a transform domain are executed using the at least one photonic processor.