Convolution streaming engine for deep neural networks

1. A method comprising:

providing an input corresponding to an image into a neural network for analysis of the image, the neural network including a kernel;

generating, during a convolution operation of the neural network, an indexing matrix that includes multiple panel matrices by mapping elements within a window of a feature map representing the input onto a portion of the indexing matrix, wherein the multiple panel matrices are based on different portions of the feature map; and

successively combining each of the multiple panel matrices with the kernel to generate an output of the convolution operation.

2. The method of claim 1 , wherein generating the indexing matrix comprises:

mapping initial elements within the window of the feature map onto a column of the indexing matrix, wherein a size of the window corresponds to a size of the kernel;

shifting the window such that a portion of the window includes at least one element included in a previous mapping and at least one new element not included in any previous mapping;

after the window is shifted, mapping current elements within the window onto a sequential neighboring column of the indexing matrix;

reshuffling the feature map based on the indexing matrix; and

partitioning the reshuffled feature map into the multiple panel matrices; and

wherein shifting of the window and mapping of the current elements within the window onto the sequential neighboring column continues until each element of the feature map is mapped.

3. The method of claim 2 , wherein shifting the window comprises:

shifting the window across the feature map in a horizontal direction from a first location;

after the window reaches an edge of the feature map, returning the window to the first location and shifting the window in a vertical direction; and

after the window is shifted in the vertical direction, shifting the window across the feature map in the horizontal direction; and

wherein the window continues to shift horizontally and vertically until each element of the feature map is mapped onto the indexing matrix.

4. The method of claim 2 , wherein:

mapping the initial elements or the current elements within the window comprises mapping each element within the window from left to right and top to bottom onto one column of the indexing matrix; and

shifting the window comprises shifting the window from a first location left to right and top to bottom.

5. The method of claim 2 , wherein the indexing matrix is a function of the size of the kernel, a size of the feature map, a padding, and a stride.

6. The method of claim 1 , wherein the multiple panel matrices are sized based on a size of the kernel and a compute capacity of the neural network.

7. The method of claim 1 , wherein:

the output comprises an output matrix; and

successively combining each of the multiple panel matrices with the kernel to generate the output comprises:

combining a first panel matrix of the multiple panel matrices with the kernel to generate a first value;

storing the first value in a first cell of the output matrix;

combining each sequential one of the multiple panel matrices with the kernel to generate sequential values; and

storing each of the sequential values in a neighboring cell of the output matrix.

8. An electronic device comprising:

at least one processor configured to:

provide an input corresponding to an image into a neural network for analysis of the image, the neural network including a kernel;

generate, during a convolution operation of the neural network, an indexing matrix that includes, multiple panel matrices, wherein, to generate the indexing matrix, the at least one processor is configured to map elements within a window of a feature map representing the input onto a portion of the indexing matrix, and wherein the multiple panel matrices are based on different portions of the feature map; and

successively combine each of the multiple panel matrices with the kernel to generate an output of the convolution operation.

9. The electronic device of claim 8 , wherein, to generate the indexing matrix, the at least one processor is configured to:

map initial elements within the window of the feature map onto a column of the indexing matrix, wherein a size of the window corresponds to a size of the kernel;

shift the window such that a portion of the window includes at least one element included in a previous mapping and at least one new element not included in any previous mapping;

after the window is shifted, map current elements within the window onto a sequential neighboring column of the indexing matrix;

reshuffle the feature map based on the indexing matrix;

partition the reshuffled feature map into the multiple panel matrices; and

continue to shift the window and map the current elements within the window onto the sequential neighboring column until each element of the feature map is mapped.

10. The electronic device of claim 9 , wherein, to shift the window, the at least one processor is configured to:

shift the window across the feature map in a horizontal direction from a first location;

after the window reaches an edge of the feature map, return the window to the first location and shift the window in a vertical direction;

after the window is shifted in the vertical direction, shift the window across the feature map in the horizontal direction; and

continue to shift the window horizontally and vertically until each element of the feature map is mapped onto the indexing matrix.

11. The electronic device of claim 9 , wherein:

to map the initial elements or the current elements within the window, the at least one processor is configured to map each element within the window from left to right and top to bottom onto one column of the indexing matrix; and

to shift the window, the at least one processor is configured to shift the window from a first location left to right and top to bottom.

12. The electronic device of claim 9 , wherein the indexing matrix is a function of the size of the kernel, a size of the feature map, a padding, and a stride.

13. The electronic device of claim 8 , wherein the multiple panel matrices are sized based on a size of the kernel and a compute capacity of the neural network.

14. The electronic device of claim 8 , wherein:

the output comprises an output matrix; and

to successively combine each of the multiple panel matrices with the kernel to generate the output, the at least one processor is configured to:

combine a first panel matrix of the multiple panel matrices with the kernel to generate a first value;

store the first value in a first cell of the output matrix;

combine each sequential one of the multiple panel matrices with the kernel to generate sequential values; and

store each of the sequential values in a neighboring cell of the output matrix.

15. A non-transitory computer readable medium containing computer readable program code that, when executed by at least one processor of an electronic device, causes the at least one processor to:

provide an input corresponding to an image into a neural network for analysis of the image, the neural network including a kernel;

generate, during a convolution operation of the neural network, an indexing matrix that includes multiple panel matrices, wherein the computer readable program code that when executed causes the at least one processor to generate the indexing matrix comprises computer readable program code that when executed causes the at least one processor to map elements within a window of a feature map representing the input onto a portion of the indexing matrix, and wherein the multiple panel matrices are based on different portions of the feature map; and

successively combine each of the multiple panel matrices with the kernel to generate an output of the convolution operation.

16. The non-transitory computer readable medium of claim 15 , wherein the computer readable program code that when executed causes the at least one processor to generate the indexing matrix comprises computer readable program code that when executed causes the at least one processor to:

map initial elements within the window of the feature map onto a column of the indexing matrix, wherein a size of the window corresponds to a size of the kernel;

shift the window such that a portion of the window includes at least one element included in a previous mapping and at least one new element not included in any previous mapping;

after the window is shifted, map current elements within the window onto a sequential neighboring column of the indexing matrix;

reshuffle the feature map based on the indexing matrix;

partition the reshuffled feature map into the multiple panel matrices; and

continue to shift the window and map the current elements within the window onto the sequential neighboring column until each element of the feature map is mapped.

17. The non-transitory computer readable medium of claim 16 , wherein the computer readable program code that when executed causes the at least one processor to shift the window comprises computer readable program code that when executed causes the at least one processor to:

shift the window across the feature map in a horizontal direction from a first location;

after the window reaches an edge of the feature map, return the window to the first location and shift the window in a vertical direction;

after the window is shifted in the vertical direction, shift the window across the feature map in the horizontal direction; and

continue to shift the window horizontally and vertically until each element of the feature map is mapped onto the indexing matrix.

18. The non-transitory computer readable medium of claim 16 , wherein:

the computer readable program code that when executed causes the at least one processor to map the initial elements or the current elements within the window comprises computer readable program code that when executed causes the at least one processor to map each element within the window from left to right and top to bottom onto one column of the indexing matrix; and

the computer readable program code that when executed causes the at least one processor to shift the window comprises computer readable program code that when executed causes the at least one processor to shift the window from a first location left to right and top to bottom.

19. The non-transitory computer readable medium of claim 15 , wherein the multiple panel matrices are sized based on a size of the kernel and a compute capacity of the neural network.

20. The non-transitory computer readable medium of claim 15 , wherein:

the output comprises an output matrix; and

the computer readable program code that when executed causes the at least one processor to successively combine each of the multiple panel matrices with the kernel to generate the output comprises computer readable program code that when executed causes the at least one processor to:

combine a first panel matrix of the multiple panel matrices with the kernel to generate a first value;

store the first value in a first cell of the output matrix;

combine each sequential one of the multiple panel matrices with the kernel to generate sequential values; and

store each of the sequential values in a neighboring cell of the output matrix.