Convolution with kernel expansion and tensor accumulation
Certain aspects of the present disclosure provide techniques for kernel expansion. An input data tensor is received at a first layer in a neural network, and a first convolution is performed for a first kernel, where the first kernel has a size greater than a preferred size. Performing the first convolution comprises generating a plurality of intermediate tensors by performing a plurality of intermediate convolutions using a plurality of intermediate kernels with a size of the preferred size, and accumulating the plurality of intermediate tensors to generate an output tensor for the first convolution.
1 . A computer-implemented method of machine learning, comprising:
receiving, at a first layer in a neural network, an input data tensor; and
performing a first convolution for a first kernel in the first layer in the neural network, wherein the first kernel has a size greater than a preferred size, comprising:
generating a plurality of intermediate tensors by performing a plurality of intermediate convolutions using the input data tensor and a plurality of intermediate kernels with a size of the preferred size, wherein an intermediate kernel of the plurality of intermediate kernels comprises a portion of the first kernel and has the same number of dimensions as the first kernel; and
accumulating the plurality of intermediate tensors to generate an output tensor for the first convolution.
2 . The method of claim 1 , wherein performing the first convolution for the first kernel comprises:
performing a first intermediate convolution on a set of elements in a central region in the input data tensor; and
performing a second intermediate convolution on a set of elements on a perimeter of the input data tensor.
3 . The method of claim 1 , further comprising:
prior to accumulating the plurality of intermediate tensors, volume shifting each of the plurality of intermediate tensors.
4 . The method of claim 1 , wherein accumulating the plurality of intermediate tensors comprises at least one of:
performing element-wise summation of the plurality of intermediate tensors, or
performing pointwise accumulation on the plurality of intermediate tensors.
5 . The method of claim 1 , wherein the preferred size corresponds to one or more hardware components of a computer.
6 . The method of claim 1 , wherein:
at least a first intermediate kernel of the plurality of intermediate kernels includes one or more empty elements, and
performing an intermediate convolution using the first intermediate kernel comprises refraining from processing each of the one or more empty elements.
7 . The method of claim 1 , wherein performing the first convolution for the first kernel further comprises:
applying a space-to-depth technique to generate the plurality of intermediate tensors by reducing a spatial dimensionality of the input data tensor and increasing a depth of the input data tensor based on a depth expansion ratio r.
8 . The method of claim 7 , wherein:
the first convolution is performed using a stride equal to s, wherein s is divisible by r; and
the plurality of intermediate convolutions are performed using a stride equal
s
r
.
9 . The method of claim 7 , wherein:
the first convolution is performed using a stride equal to s, wherein s is not divisible by r; and
the plurality of intermediate convolutions are performed using a fixed reshuffling or re-addressing of channels in the plurality of intermediate tensors.
10 . The method of claim 1 , wherein performing the first convolution for the first kernel is performed using in-step accumulation, comprising:
generating a plurality of intermediate tensors by performing a plurality of intermediate in-step convolutions;
performing in-step activation shifting of each of the plurality of intermediate tensors; and
performing in-step activation accumulation of the plurality of shifted intermediate tensors.
11 . A processing system comprising:
at least one memory comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the processing system to perform an operation comprising:
receiving, at a first layer in a neural network, an input data tensor; and
performing a first convolution for a first kernel in the first layer in the neural network, wherein the first kernel has a size greater than a preferred size, comprising:
generating a plurality of intermediate tensors by performing a plurality of intermediate convolutions using the input data tensor and a plurality of intermediate kernels with a size of the preferred size, wherein an intermediate kernel of the plurality of intermediate kernels comprises a portion of the first kernel and has the same number of dimensions as the first kernel; and
accumulating the plurality of intermediate tensors to generate an output tensor for the first convolution.
12 . The processing system of claim 11 , wherein performing the first convolution for the first kernel comprises:
performing a first intermediate convolution on a set of elements in a central region in the input data tensor; and
performing a second intermediate convolution on a set of elements on a perimeter of the input data tensor.
13 . The processing system of claim 11 , the operation further comprising:
prior to accumulating the plurality of intermediate tensors, volume shifting each of the plurality of intermediate tensors.
14 . The processing system of claim 11 , wherein accumulating the plurality of intermediate tensors comprises performing element-wise summation of the plurality of intermediate tensors.
15 . The processing system of claim 11 , wherein accumulating the plurality of intermediate tensors comprises performing pointwise accumulation on the plurality of intermediate tensors.
16 . The processing system of claim 11 , wherein:
at least a first intermediate kernel of the plurality of intermediate kernels includes one or more empty elements, and
performing an intermediate convolution using the first intermediate kernel comprises refraining from processing each of the one or more empty elements.
17 . The processing system of claim 11 , wherein performing the first convolution for the first kernel further comprises:
applying a space-to-depth technique to generate the plurality of intermediate tensors by reducing a spatial dimensionality of the input data tensor and increasing a depth of the input data tensor based on a depth expansion ratio r.
18 . The processing system of claim 17 , wherein:
the first convolution is performed using a stride equal to s, wherein s is divisible by r; and
the plurality of intermediate convolutions are performed using a stride equal to
s
r
.
19 . The processing system of claim 17 , wherein:
the first convolution is performed using a stride equal to s, wherein s is not divisible by r; and
the plurality of intermediate convolutions are performed using a fixed reshuffling or re-addressing of channels in the plurality of intermediate tensors.
20 . The processing system of claim 11 , wherein performing the first convolution for the first kernel is performed using in-step accumulation, comprising:
generating a plurality of intermediate tensors by performing a plurality of intermediate in-step convolutions;
performing in-step activation shifting of each of the plurality of intermediate tensors; and
performing in-step activation accumulation of the plurality of shifted intermediate tensors.
21 . A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform an operation comprising:
receiving, at a first layer in a neural network, an input data tensor; and
performing a first convolution for a first kernel in the first layer in the neural network, wherein the first kernel has a size greater than a preferred size, comprising:
generating a plurality of intermediate tensors by performing a plurality of intermediate convolutions using the input data tensor and a plurality of intermediate kernels with a size of the preferred size, wherein an intermediate kernel of the plurality of intermediate kernels comprises a portion of the first kernel and has the same number of dimensions as the first kernel; and
accumulating the plurality of intermediate tensors to generate an output tensor for the first convolution.
22 . The non-transitory computer-readable medium of claim 21 , wherein performing the first convolution for the first kernel comprises:
performing a first intermediate convolution on a set of elements in a central region in the input data tensor; and
performing a second intermediate convolution on a set of elements on a perimeter of the input data tensor.
23 . The non-transitory computer-readable medium of claim 21 , the operation further comprising:
prior to accumulating the plurality of intermediate tensors, volume shifting each of the plurality of intermediate tensors.
24 . The non-transitory computer-readable medium of claim 21 , wherein accumulating the plurality of intermediate tensors comprises performing element-wise summation of the plurality of intermediate tensors.
25 . The non-transitory computer-readable medium of claim 21 , wherein accumulating the plurality of intermediate tensors comprises performing pointwise accumulation on the plurality of intermediate tensors.
26 . The non-transitory computer-readable medium of claim 21 , wherein:
at least a first intermediate kernel of the plurality of intermediate kernels includes one or more empty elements, and
performing an intermediate convolution using the first intermediate kernel comprises refraining from processing each of the one or more empty elements.
27 . The non-transitory computer-readable medium of claim 21 , wherein performing the first convolution for the first kernel further comprises:
applying a space-to-depth technique to generate the plurality of intermediate tensors by reducing a spatial dimensionality of the input data tensor and increasing a depth of the input data tensor based on a depth expansion ratio r.
28 . The non-transitory computer-readable medium of claim 27 , wherein:
the first convolution is performed using a stride equal to s, wherein s is divisible by r; and
the plurality of intermediate convolutions are performed using a stride equal to
s
r
.
29 . The non-transitory computer-readable medium of claim 21 , wherein performing the first convolution for the first kernel is performed using in-step accumulation, comprising:
generating a plurality of intermediate tensors by performing a plurality of intermediate in-step convolutions;
performing in-step activation shifting of each of the plurality of intermediate tensors; and
performing in-step activation accumulation of the plurality of shifted intermediate tensors.
30 . A processing system comprising:
means for receiving, at a layer in a neural network, an input data tensor; and
means for performing a convolution for a first kernel in the layer in the neural network, wherein the first kernel has a size greater than a preferred size, the means for performing comprising:
means for generating a plurality of intermediate tensors by performing a plurality of intermediate convolutions using the input data tensor and a plurality of intermediate kernels with a size of the preferred size, wherein an intermediate kernel of the plurality of intermediate kernels comprises a portion of the first kernel and has the same number of dimensions as the first kernel; and
means for accumulating the plurality of intermediate tensors to generate an output tensor for the convolution.