Computing convolutions using a neural network processor

1. A method for computing a layer output for a convolutional neural network layer from a layer input, the layer output being computed using a neural network hardware circuit, the method comprising:

obtaining, using the neural network hardware circuit, the layer input represented by an input matrix having at least a first dimension, a second dimension, and a third dimension, wherein the third dimension of the input matrix comprises a plurality of depth levels;

deriving, using the neural network hardware circuit and for the third dimension of the input matrix, M two-dimensional matrices by splitting up respective sets of inputs of the input matrix to form a two-dimensional matrix for each depth level of the plurality of depth levels, wherein M is a positive integer representing a respective number of depth levels of the third dimension, and wherein M is greater than one;

concurrently processing, using a hardware systolic array of the neural network hardware circuit, two or more two-dimensional matrices of the M two-dimensional matrices in response to sending a respective set of inputs for each two-dimensional matrix to one or more rows of the hardware systolic array to generate one or more accumulated values; and

generating, using the hardware systolic array of the neural network hardware circuit, the layer output based on the one or more accumulated values.

2. The method of claim 1 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices comprises concurrently processing two or more two-dimensional matrices of the M two-dimensional matrices using a plurality of cells included in the hardware systolic array.

3. The method of claim 2 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices comprises processing each two-dimensional matrix of the two or more two-dimensional matrices using a different subdivision of cells of the plurality of cells included in the hardware systolic array.

4. The method of claim 2 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

at a given clock cycle, storing, in a first cell of the hardware systolic array, a first element of a first two-dimensional matrix of the two or more two-dimensional matrices; and

at a subsequent clock cycle, shifting the first element in the first cell to a second cell that is adjacent to the first cell and storing a second element of the first two-dimensional matrix in the first cell.

5. The method of claim 2 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

obtaining weight inputs; and

generating the one or more accumulated values based on the weight inputs and activation inputs in the two or more two-dimensional matrices.

6. The method of claim 5 ,

wherein the convolutional neural network layer includes a plurality of kernels that include a plurality of sets of weight inputs,

wherein obtaining the weight inputs comprises receiving the plurality of kernels.

7. The method of claim 5 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

at a given clock cycle, storing, in a first cell of the hardware systolic array, a first weight input of the weight inputs; and

at a subsequent clock cycle, shifting the first weight input in the first cell to a second cell that is adjacent to the first cell and storing a second weight input of the weight inputs in the first cell.

8. The method of claim 5 , wherein generating the one or more accumulated values comprises computing a dot product of a weight input and an activation input that are stored in the same cell.

9. The method of claim 1 , wherein:

the third dimension of the input matrix represents a feature dimension,

the first and the second dimensions of the input matrix represent spatial dimensions that correspond to spatial coordinates of an input image, and

each depth level of the plurality of depth levels of the feature dimension corresponds to a respective color feature of the feature dimension.

10. The method of claim 9 , wherein deriving the M two-dimensional matrices comprises determining that the third dimension of the input matrix represents the feature dimension.

11. A system for computing a layer output for a convolutional neural network layer from a layer input, the system comprising circuitry configured to:

obtain, using a neural network hardware circuit of the system, the layer input represented by an input matrix having at least a first dimension, a second dimension, and a third dimension, wherein the third dimension of the input matrix comprises a plurality of depth levels;

derive, using the neural network hardware circuit of the system and for the third dimension of the input matrix, M two-dimensional matrices by splitting up respective sets of inputs of the input matrix to form a two-dimensional matrix for each depth level of the plurality of depth levels, wherein M is a positive integer representing a respective number of depth levels of the third dimension, and wherein M is greater than one;

concurrently process, using a hardware systolic array of the neural network hardware circuit, two or more two-dimensional matrices of the M two-dimensional matrices in response to sending a respective set of inputs for each two-dimensional matrix to one or more rows of the hardware systolic array to generate one or more accumulated values; and

generate, using the hardware systolic array of the neural network hardware circuit, the layer output based on the one or more accumulated values.

12. The system of claim 11 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices comprises concurrently processing two or more two-dimensional matrices of the M two-dimensional matrices using a plurality of cells included in the hardware systolic array.

13. The system of claim 12 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices comprises processing each two-dimensional matrix of the two or more two-dimensional matrices using a different subdivision of cells of the plurality of cells included in the hardware systolic array.

14. The system of claim 12 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

at a given clock cycle, storing, in a first cell of the hardware systolic array, a first element of a first two-dimensional matrix of the two or more two-dimensional matrices; and

at a subsequent clock cycle, shifting the first element in the first cell to a second cell that is adjacent to the first cell and storing a second element of the first two-dimensional matrix in the first cell.

15. The system of claim 12 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

obtaining weight inputs; and

generating the one or more accumulated values based on the weight inputs and activation inputs in the two or more two-dimensional matrices.

16. The method of claim 15 ,

wherein the convolutional neural network layer includes a plurality of kernels that include a plurality of sets of weight inputs,

wherein obtaining the weight inputs comprises receiving the plurality of kernels.

17. The method of claim 15 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

at a given clock cycle, storing, in a first cell of the hardware systolic array, a first weight input of the weight inputs; and

at a subsequent clock cycle, shifting the first weight input in the first cell to a second cell that is adjacent to the first cell and storing a second weight input of the weight inputs in the first cell.

18. The method of claim 15 , wherein generating the one or more accumulated values comprises computing a dot product of a weight input and an activation input that are stored in the same cell.

19. The method of claim 11 , wherein:

the third dimension of the input matrix represents a feature dimension,

the first and the second dimensions of the input matrix represent spatial dimensions that correspond to spatial coordinates of an input image, and

each depth level of the plurality of depth levels of the feature dimension corresponds to a respective color feature of the feature dimension.

20. A computer-readable medium having instructions stored thereon, which, when executed by one or more computers, cause the one or more computers to perform operations for computing a layer output for a convolutional neural network layer from a layer input, the operations comprising:

obtaining, using a neural network hardware circuit, the layer input represented by an input matrix having at least a first dimension, a second dimension, and a third dimension, wherein the third dimension of the input matrix comprises a plurality of depth levels;

deriving, using the neural network hardware circuit and for the third dimension of the input matrix, M two-dimensional matrices by splitting up respective sets of inputs of the input matrix to form a two-dimensional matrix for each depth level of the plurality of depth levels, wherein M is a positive integer representing a respective number of depth levels of the third dimension, and wherein M is greater than one;

concurrently processing, using a hardware systolic array of the neural network hardware circuit, two or more two-dimensional matrices of the M two-dimensional matrices in response to sending a respective set of inputs for each two-dimensional matrix to one or more rows of the hardware systolic array to generate one or more accumulated values; and

generating, using the hardware systolic array of the neural network hardware circuit, the layer output based on the one or more accumulated values.

21. The computer-readable medium of claim 20 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices comprises concurrently processing two or more two-dimensional matrices of the M two-dimensional matrices using a plurality of cells included in the hardware systolic array.

22. The computer-readable medium of claim 21 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices comprises processing each two-dimensional matrix of the two or more two-dimensional matrices using a different subdivision of cells of the plurality of cells included in the hardware systolic array.

23. The computer-readable medium of claim 21 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

at a given clock cycle, storing, in a first cell of the hardware systolic array, a first element of a first two-dimensional matrix of the two or more two-dimensional matrices; and

at a subsequent clock cycle, shifting the first element in the first cell to a second cell that is adjacent to the first cell and storing a second element of the first two-dimensional matrix in the first cell.

24. The computer-readable medium of claim 21 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

obtaining weight inputs; and

generating the one or more accumulated values based on the weight inputs and activation inputs in the two or more two-dimensional matrices.

25. The computer-readable medium of claim 24 ,

wherein the convolutional neural network layer includes a plurality of kernels that include a plurality sets of weight inputs,

wherein obtaining the weight inputs comprises receiving the plurality of kernels.

26. The computer-readable medium of claim 24 , wherein concurrently processing the two or more two-dimensional matrices of the M two-dimensional matrices to generate one or more accumulated values comprises:

at a given clock cycle, storing, in a first cell of the hardware systolic array, a first weight input of the weight inputs; and

at a subsequent clock cycle, shifting the first weight input in the first cell to a second cell that is adjacent to the first cell and storing a second weight input of the weight inputs in the first cell.

27. The computer-readable medium of claim 20 , wherein:

the third dimension of the input matrix represents a feature dimension,

the first and the second dimensions of the input matrix represent spatial dimensions that correspond to spatial coordinates of an input image, and

each depth level of the plurality of depth levels of the feature dimension corresponds to a respective color feature of the feature dimension.