Spatial frequency splitting for convolutional neural networks
View Patent ↗A system may allocate, in response to receiving an input tensor of a predetermined shape, a portion of convolution kernels to a predetermined number of spatial frequency bands; replace the convolution layer in a convolutional neural network with a sequence of band splitting, parallel convolutions, and concatenation; and form, in response to completing stages for the predetermined number of spatial frequency bands, a final output tensor.
1 . A system comprising:
a memory storing program instructions; and
a processor in communication with the memory, the processor being configured to execute the program instructions to perform processes comprising:
allocating, in response to receiving an input tensor of a predetermined shape, a portion of convolution kernels to a predetermined number of spatial frequency bands;
replacing a convolution layer in a convolutional neural network with a sequence of band splitting, parallel convolutions, and concatenation; and
forming, in response to completing stages for the predetermined number of spatial frequency bands, a final output tensor.
2 . The system of claim 1 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
splitting the input tensor for the convolution layer into different spatial spectral bands using a trained spatial frequency filter for the stage of the convolutional neural network to exploit frequency features while maintaining information of the input tensor.
3 . The system of claim 2 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
allocating, for the spatial spectral band, parallel convolutions as one of a set of convolution layers to a respective spatial spectral band.
4 . The system of claim 3 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
generating an output from the parallel convolutions.
5 . The system of claim 4 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
stacking the output into a single tensor.
6 . The system of claim 5 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
forming, based on the single tensor using concatenation along a channel axis, a concatenated tensor.
7 . The system of claim 6 , wherein the memory stores further program instructions, and wherein the processor is configured to execute the further program instructions to perform the processes further comprising:
passing the concatenated tensor to a next stage, including a pooling layer, and a non-linearity layer;
classifying an object in an input image related to the input tensor; and
storing the classification as metadata for the input image.
8 . A method comprising:
allocating, in response to receiving an input tensor of a predetermined shape, a portion of convolution kernels to a predetermined number of spatial frequency bands;
replacing a convolution layer in a convolutional neural network with a sequence of band splitting, parallel convolutions, and concatenation; and
forming, in response to completing stages for the predetermined number of spatial frequency bands, a final output tensor.
9 . The method of claim 8 , wherein the method further comprises:
splitting the input tensor for the convolution layer into different spatial spectral bands using a trained spatial frequency filter for the stage of the convolutional neural network to exploit frequency features while maintaining information of the input tensor.
10 . The method of claim 9 , wherein the method further comprises:
allocating, for the spatial spectral band, parallel convolutions as one of a set of convolution layers to a respective spatial spectral band.
11 . The method of claim 10 , wherein the method further comprises:
generating an output from the parallel convolutions.
12 . The method of claim 11 , wherein the method further comprises:
stacking the output into a single tensor.
13 . The method of claim 12 , wherein the method further comprises:
forming, based on the single tensor using concatenation along a channel axis, a concatenated tensor.
14 . The method of claim 13 , wherein the method further comprises:
passing the concatenated tensor to a next stage, including a pooling layer, and a non-linearity layer;
classifying an object in an input image related to the input tensor; and
storing the classification as metadata for the input image.
15 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method, the method comprising:
allocating, in response to receiving an input tensor of a predetermined shape, a portion of convolution kernels to a predetermined number of spatial frequency bands;
replacing a convolution layer in a convolutional neural network with a sequence of band splitting, parallel convolutions, and concatenation; and
forming, in response to completing stages for the predetermined number of spatial frequency bands, a final output tensor.
16 . The computer program product of claim 15 , further comprising additional program instructions stored on the computer readable storage medium and configured to cause the processor to perform the method further comprising:
splitting the input tensor for the convolution layer into different spatial spectral bands using a trained spatial frequency filter for the stage of the convolutional neural network to exploit frequency features while maintaining information of the input tensor.
17 . The computer program product of claim 16 , further comprising additional program instructions stored on the computer readable storage medium and configured to cause the processor to perform the method further comprising:
allocating, for the spatial spectral band, parallel convolutions as one of a set of convolution layers to a respective spatial spectral band.
18 . The computer program product of claim 17 , further comprising additional program instructions stored on the computer readable storage medium and configured to cause the processor to perform the method further comprising:
generating an output from the parallel convolutions.
19 . The computer program product of claim 18 , further comprising additional program instructions stored on the computer readable storage medium and configured to cause the processor to perform the method further comprising:
stacking the output into a single tensor.
20 . The computer program product of claim 19 , further comprising additional program instructions stored on the computer readable storage medium and configured to cause the processor to perform the method further comprising:
forming, based on the single tensor using concatenation along a channel axis, a concatenated tensor;
passing the concatenated tensor to a next stage, including a pooling layer, and a non-linearity layer;
classifying an object in an input image related to the input tensor; and
storing the classification as metadata for the input image.