Memory efficient convolution operations in deep learning neural networks
View Patent ↗Input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows is received. The input image data is transformed into a plurality of input rows. The pixel values in each input row correspond to the pixel values in a predetermined subset of the columns of the input image data and all of the rows of each column of the subset of columns. A plurality of subsets of pixel values in the plurality of input rows is determined. The number of pixel values in each row of a subset of pixel values equal in number to a number of filter values in a filter. Each input row of each subset of pixel values is convolved with the filter values of the filter to determine a corresponding output value and stored in a memory.
1. A method comprising:
receiving input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows;
selecting a plurality of subsets of the input image data, each subset comprising a plurality of adjacent columns of the input image data, each subset comprising all rows of the input image data, a number of columns in the plurality of adjacent columns equal to a predetermined width of a two-dimensional filter matrix;
arranging, in a left-to-right and top-to-bottom manner, each of the plurality of subsets into a corresponding linearized input row;
storing the plurality of linearized input rows in a memory;
convolving a first portion of a first linearized input row in the plurality of linearized input rows with filter values of the two-dimensional filter matrix to determine a corresponding first output value, a size of the first portion equal to a size of the two-dimensional filter matrix;
convolving a second portion of the first linearized input row with filter values of the two-dimensional filter matrix to determine a corresponding second output value, a size of the second portion equal to the size of the two-dimensional filter matrix, the second portion determined by shifting within the first linearized input row by a predetermined number of shift values from the first portion; and
storing each output value in the memory in a two-dimensional matrix form.
2. The method of claim 1 , wherein selecting the plurality of subsets comprises:
determining a first subset; and
determining a second subset, the second subset being determined by shifting within each input row by a predetermined number of columns.
3. The method of claim 1 , wherein convolving each linearized input row with the filter values of the two-dimensional filter matrix comprises:
multiplying each pixel value in the linearized input row by a corresponding filter value in the two-dimensional filter matrix; and
summing the results of each multiplication to determine the corresponding output value.
4. The method of claim 3 , further comprising: arranging the filter values of the two-dimensional filter matrix in a column representation prior to the convolving.
5. The method of claim 4 , further comprising transforming the filter values from a two-dimensional matrix representation to the column representation.
6. The method of claim 1 , wherein the convolving is performed in a neural network.
7. A computer usable program product comprising one or more computer-readable storage media, and program instructions stored on at least one of the one or more computer readable storage media, the stored program instructions comprising:
program instructions to receive input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows;
program instructions to select a plurality of subsets of the input image data, each subset comprising a plurality of adjacent columns of the input image data, each subset comprising all rows of the input image data, a number of columns in the plurality of adjacent columns equal to a predetermined width of a two-dimensional filter matrix;
program instructions to arrange, in a left-to-right and top-to-bottom manner, each of the plurality of subsets into a corresponding linearized input row;
program instructions to store the plurality of linearized input rows in a memory;
program instructions to convolve a first portion of a first linearized input row in the plurality of linearized input rows with filter values of the two-dimensional filter matrix to determine a corresponding first output value, a size of the first portion equal to a size of the two-dimensional filter matrix;
program instructions to convolve a second portion of the first linearized input row with filter values of the two-dimensional filter matrix to determine a corresponding second output value, a size of the second portion equal to the size of the two-dimensional filter matrix, the second portion determined by shifting within the first linearized input row by a predetermined number of shift values from the first portion; and
program instructions to store each output value in the memory in a two-dimensional matrix form.
8. The computer usable program product of claim 7 , wherein program instructions to select the plurality of subsets further comprises:
program instructions to determine a first subset; and
program instructions to determine a second subset, the second subset being determined by shifting within each input row by a predetermined number of columns.
9. The computer usable program product of claim 7 , wherein the program instructions to convolve each linearized input row with the filter values of the two-dimensional filter matrix comprise:
program instructions to multiply each pixel value in the linearized input row by a corresponding filter value in the two-dimensional filter matrix; and
program instructions to sum the results of each multiplication to determine the corresponding output value.
10. The computer usable program product of claim 9 , further comprising:
program instructions to arrange the filter values of the two-dimensional filter matrix in a column representation prior to the convolving.
11. The computer usable program product of claim 10 , further comprising program instructions to transform the filter values from a two-dimensional matrix representation to the column representation.
12. A computer system comprising: one or more processors, one or more computer-readable memories, and one or more computer-readable storage media, and program instructions stored on at least one of the one or more computer-readable storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, the stored program instructions comprising:
program instructions to receive input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows;
program instructions to select a plurality of subsets of the input image data, each subset comprising a plurality of adjacent columns of the input image data, each subset comprising all rows of the input image data, a number of columns in the plurality of adjacent columns equal to a predetermined width of a two-dimensional filter matrix;
program instructions to arrange, in a left-to-right and top-to-bottom manner, each of the plurality of subsets into a corresponding linearized input row;
program instructions to store the plurality of linearized input rows in a memory;
program instructions to convolve a first portion of a first linearized input row in the plurality of linearized input rows with filter values of the two-dimensional filter matrix to determine a corresponding first output value, a size of the first portion equal to a size of the two-dimensional filter matrix;
program instructions to convolve a second portion of the first linearized input row with filter values of the two-dimensional filter matrix to determine a corresponding second output value, a size of the second portion equal to the size of the two-dimensional filter matrix, the second portion determined by shifting within the first linearized input row by a predetermined number of shift values from the first portion; and
program instructions to store each output value in the memory in a two-dimensional matrix form.
13. The computer system of claim 12 , wherein program instructions to select the plurality of subsets further comprises:
program instructions to determine a first subset; and
program instructions to determine a second subset, the second subset being determined by shifting within each input row by a predetermined number of columns.
14. The computer system of claim 12 , wherein the program instructions to convolve each linearized input row with the filter values of the two-dimensional filter matrix comprise:
program instructions to multiply each pixel value in the linearized input row by a corresponding filter value in the two-dimensional filter matrix; and
program instructions to sum the results of each multiplication to determine the corresponding output value.
15. The method of claim 1 , wherein the predetermined number of shift values comprises a product of the predetermined width of the two-dimensional filter matrix and a filter slide value.
16. The computer usable program product of claim 7 , wherein the predetermined number of shift values comprises a product of the predetermined width of the two-dimensional filter matrix and a filter slide value.
17. The computer system of claim 12 , wherein the predetermined number of shift values comprises a product of the predetermined width of the two-dimensional filter matrix and a filter slide value.