Fine-grained sparsity computations in systolic array
A systolic array can implement an architecture tailored to perform matrix multiplications on constrained fine-grained sparse weight matrices. Each processing element in the systolic array may include a weight register configured to store a weight value, and a multiplexor configured to select a feature map (FMAP) input element from multiple FMAP input data buses based on metadata associated with the weight value. Each processing element may also include a multiplier configured to multiply the selected feature map input element with the weight value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.
1. A neural network processor comprising:
a memory operable to store a constrained fine-grained sparse weight matrix as a set of non-zero weight values and metadata associated with each non-zero weight value indicating which row in the constrained fine-grained sparse weight matrix that the non-zero weight value corresponds to; and
an array of processing elements arranged in rows and columns, wherein each row of processing elements has a plurality of feature map (FMAP) row input buses, wherein the array of processing elements is operable in a sparsity mode of operation and a normal mode of operation, and wherein the array of processing elements is operable to perform a sparse matrix multiplication operation on the constrained fine-grained sparse weight matrix by:
loading the set of non-zero weight values into a row of the array of processing elements; and
for each processing element in the row of the array of processing elements:
selecting a FMAP input element from the plurality of FMAP row input buses based on the metadata associated with the non-zero weight value loaded into the processing element;
multiplying the selected FMAP input element with the non-zero weight value loaded into the processing element to generate a multiplication result;
adding the multiplication result to a column input to generate a partial sum; and
outputting the partial sum to a column output.
2. The neural network processor of claim 1 , wherein the array of processing elements is operable to concurrently perform multiple sparse matrix multiplication operations using multiple rows of the array of processing elements.
3. The neural network processor of claim 1 , wherein the array of processing elements is operable to turn off unused rows of the array of processing elements while performing the sparse matrix multiplication operation.
4. The neural network processor of claim 1 , wherein the array of processing elements is operable to perform a matrix multiplication operation on a dense weight matrix using every row of the array of processing elements.
5. An integrated circuit device comprising:
an array of processing elements arranged in rows and columns, wherein each processing element includes:
a weight register configured to store a weight value;
a multiplexor configured to select, based on metadata associated with the weight value, a feature map (FMAP) input element from a plurality of FMAP input data buses;
a multiplier configured to multiply the selected FMAP input element with the weight value to generate a multiplication result; and
an adder configured to add the multiplication result to a partial sum input to generate a partial sum output,
wherein the array of processing elements is operable in a sparsity mode of operation and a normal mode of operation.
6. The integrated circuit device of claim 5 , wherein in the sparsity mode of operation, each row of the array of processing elements is operable to perform a sparse matrix multiplication operation on a two-dimensional constrained fine-grained sparse weight matrix.
7. The integrated circuit device of claim 5 , wherein in the sparsity mode of operation, the metadata associated with the weight value stored in the weight register includes sparsity row information to indicate which row of a two-dimensional constrained fine-grained sparse weight matrix that the weight value corresponds to, and the multiplexor is configured to select the FMAP input element based on the sparsity row information.
8. The integrated circuit device of claim 7 , wherein the sparsity row information is encoded using one-hot encoding.
9. The integrated circuit device of claim 7 , wherein the sparsity row information is represented as a row number.
10. The integrated circuit device of claim 5 , wherein in the normal mode of operation, the array of processing elements is operable to perform a matrix multiplication operation on a weight matrix using multiple rows of the array of processing elements.
11. The integrated circuit device of claim 5 , wherein in a normal mode of operation, the multiplexor is configured to select the FMAP input element on a predetermined FMAP input data bus.
12. The integrated circuit device of claim 5 , wherein the metadata includes mode information to control whether the array of processing elements is operating in the sparsity mode of operation or in the normal mode of operation.
13. A method for operating an acceleration circuit having an array of processing elements arranged in rows and columns to perform a matrix multiplication operation, the method comprising:
receiving, by a processing element in the array of processing elements, a weight value;
receiving, by the processing element, a set of feature map (FMAP) input elements on respective FMAP input data buses;
selecting, by a multiplexer of the processing element, a FMAP input element from the FMAP input data buses based on metadata associated with the weight value;
multiplying, by a multiplier of the processing element, the selected feature map FMAP input element with the weight value to generate a multiplication result; and
adding, by an adder of the processing element, the multiplication result to a partial sum input to generate a partial sum output,
wherein the array of processing elements is operable in a sparsity mode of operations and a normal mode of operation.
14. The method of claim 13 , wherein the weight value is a non-zero weight value of a constrained fine-grained sparse weight matrix loaded from a memory that stores the constrained fine-grained sparse weight matrix as a set of non-zero values.
15. The method of claim 14 , wherein the memory further stores sparsity row information associated with each non-zero value, the sparsity row information indicating which row in the constrained fine-grained sparse weight matrix that the non-zero value corresponds to.
16. The method of claim 15 , wherein the sparsity row information is represented using one-hot encoding or using a row number.
17. The method of claim 13 , wherein the matrix multiplication operation is a sparse matrix multiplication operation being performed on a constrained fine-grained sparse weight matrix, and the sparse matrix multiplication operation is performed using one row of the array of processing elements.
18. The method of claim 17 , further comprising:
turning off unused rows of the array of processing elements while performing the sparse matrix multiplication operation.
19. The method of claim 17 , further comprising:
performing another sparse matrix multiplication operation on another constrained fine-grained sparse weight matrix using another row of the array of processing elements concurrently with the sparse matrix multiplication operation.