Computing convolutions using a neural network processor
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for computing a layer output for a convolutional neural network layer, the method comprising: receiving the layer input, the layer input comprising a plurality of activation inputs, the plurality of activation inputs represented as a multi-dimensional matrix comprising a plurality of depth levels, each depth level being a respective matrix of distinct activation inputs from the plurality of activation inputs; sending each respective kernel matrix structure to a distinct cell along a first dimension of the systolic array; for each depth level, sending the respective matrix of distinct activation inputs to a distinct cell along a second dimension of the systolic array; causing the systolic array to generate an accumulated output from the respective matrices sent to the cells; and generating the layer output from the accumulated output.
1. A method for generating an output for a layer of a neural network implemented on a hardware circuit, the method comprising:
receiving for the layer of the neural network:
layer inputs represented as a first multi-dimensional input matrix; and
a plurality of weights;
determining that computations to generate the output for the layer are convolutional computations;
converting, based on a dimension of the first multi-dimensional input matrix, the convolutional computations to two-dimensional matrix multiplications between the layer inputs and the plurality of weights, the converting comprising flattening the first multi-dimensional input matrix into a plurality of second multi-dimensional input matrices, independently shifting each of the second multi-dimensional input matrices between cells of the hardware circuit, and performing a convolution computation on the plurality of second multi-dimensional input matrices; and
generating the output in response to performing a portion of the two-dimensional matrix multiplications in parallel at the hardware circuit.
2. The method of claim 1 , wherein converting the convolutional computations to two-dimensional matrix multiplications comprises:
generating the plurality of second multi-dimensional input matrices based on a feature dimension of the first multi-dimensional input matrix.
3. The method of claim 2 , wherein generating the plurality of second multi-dimensional input matrices comprises:
deriving each of the second second multi-dimensional input matrices from the first multi-dimensional input matrix based on a depth level of the feature dimension.
4. The method of claim 2 , wherein generating the plurality of second multi-dimensional input matrices comprises:
generating each of the second multi-dimensional input matrices in response to splitting up respective sets of inputs of the first multi-dimensional input matrix to form a respective two-dimensional matrix for each of the second multi-dimensional input matrices.
5. The method of claim 2 , wherein generating the plurality of second multi-dimensional input matrices comprises:
generating a respective second multi-dimensional input matrix for each depth level of a plurality of depth levels of the feature dimension of the first multi-dimensional input matrix.
6. The method of claim 2 , wherein the plurality of weights are represented as multi-dimensional kernel matrices and the method comprises:
providing each of the second multi-dimensional input matrices to a distinct row of a hardware matrix unit located on the hardware circuit; and
providing each of the multi-dimensional kernel matrices along a column dimension of the hardware matrix unit.
7. The method of claim 6 , wherein generating the output comprises:
generating a plurality of dot products in response to performing multiple two-dimensional matrix multiplications in parallel at the hardware circuit, each two-dimensional matrix multiplication being between a respective layer input of a second multi-dimensional input matrix and a respective weight input of a multi-dimensional kernel matrix.
8. The method of claim 6 , wherein the hardware matrix unit is a two-dimensional systolic array comprising a row dimension and a column dimension.
9. The method of claim 8 , wherein each of the multi-dimensional kernel matrices represents a set of weights for a convolutional neural network layer and the method further comprises:
determining that the convolutional neural network layer comprises fewer sets of weights than columns in the two-dimensional systolic array;
in response to determining that the convolutional neural network layer comprises fewer sets of weights than columns, replicating one or more matrix structures for the sets of weights; and
sending the replicated matrix structures to unused columns of the two-dimensional systolic array to generate the output.
10. A system for generating an output for a layer of a neural network implemented on a hardware circuit, the system comprising:
one or more processing devices; and
one or more non-transitory machine-readable storage devices for storing instructions that are executable by the one or more processing devices to cause performance of operations comprising:
receiving for the layer of the neural network:
layer inputs represented as a first multi-dimensional input matrix; and
a plurality of weights;
determining that computations to generate the output for the layer are convolutional computations;
converting, based on a dimension of the first multi-dimensional input matrix, the convolutional computations to two-dimensional matrix multiplications between the layer inputs and the plurality of weights, the converting comprising flattening the first multi- dimensional input matrix into a plurality of second multi-dimensional input matrices, independently shifting each of the second multi-dimensional input matrices between cells of the hardware circuit, and performing a convolution computation on the plurality of second multi-dimensional input matrices; and
generating the output in response to performing a portion of the two-dimensional matrix multiplications in parallel at the hardware circuit.
11. The system of claim 10 , wherein converting the convolutional computations to two-dimensional matrix multiplications comprises:
generating the plurality of second multi-dimensional input matrices based on a feature dimension of the first multi-dimensional input matrix.
12. The system of claim 11 , wherein generating the plurality of second multi-dimensional input matrices comprises:
deriving each of the second second multi-dimensional input matrices from the first multi-dimensional input matrix based on a depth level of the feature dimension.
13. The system of claim 11 , wherein generating the plurality of second multi-dimensional input matrices comprises:
generating each of the second multi-dimensional input matrices in response to splitting up respective sets of inputs of the first multi-dimensional input matrix to form a respective two-dimensional matrix for each of the second multi-dimensional input matrices.
14. The system of claim 11 , wherein generating the plurality of second multi-dimensional input matrices comprises:
generating a respective second multi-dimensional input matrix for each depth level of a plurality of depth levels of the feature dimension of the first multi-dimensional input matrix.
15. The system of claim 11 , wherein the plurality of weights are represented as multi-dimensional kernel matrices and the operations comprise:
providing each of the second multi-dimensional input matrices to a distinct row of a hardware matrix unit located on the hardware circuit; and
providing each of the multi-dimensional kernel matrices along a column dimension of the hardware matrix unit.
16. The system of claim 15 , wherein generating the output comprises:
generating a plurality of dot products in response to performing multiple two-dimensional matrix multiplications in parallel at the hardware circuit, each two-dimensional matrix multiplication being between a respective layer input of a second multi-dimensional input matrix and a respective weight input of a multi-dimensional kernel matrix.
17. The system of claim 15 , wherein the hardware matrix unit is a two-dimensional systolic array comprising a row dimension and a column dimension.
18. The system of claim 17 , wherein each of the multi-dimensional kernel matrices represents a set of weights for a convolutional neural network layer and the operations further comprise:
determining that the convolutional neural network layer comprises fewer sets of weights than columns in the two-dimensional systolic array;
in response to determining that the convolutional neural network layer comprises fewer sets of weights than columns, replicating one or more matrix structures for the sets of weights; and
sending the replicated matrix structures to unused columns of the two-dimensional systolic array to generate the output.
19. A non-transitory machine-readable storage device for storing instructions that are executable by a processing device to cause performance of operations to generate an output for a layer of a neural network implemented on a hardware circuit, the operations comprising:
receiving for the layer of the neural network:
layer inputs represented as a first multi-dimensional input matrix; and
a plurality of weights;
determining that computations to generate the output for the layer are convolutional computations;
converting, based on a dimension of the first multi-dimensional input matrix, the convolutional computations to two-dimensional matrix multiplications between the layer inputs and the plurality of weights, the converting comprising flattening the first multi-dimensional input matrix into a plurality of second multi-dimensional input matrices, independently shifting each of the second multi-dimensional input matrices between cells of the hardware circuit, and performing a convolution computation on the plurality of second multi-dimensional input matricesflattened matrix; and
generating the output in response to performing a portion of the two-dimensional matrix multiplications in parallel at the hardware circuit.
20. The machine-readable storage devices of claim 19 , wherein converting the convolutional computations to two-dimensional matrix multiplications comprises:
generating the plurality of second multi-dimensional input matrices based on a feature dimension of the first multi-dimensional input matrix.