Emulating fine-grained sparsity in a systolic array
To take advantage of the architecture of a systolic array tailored to perform sparse matrix multiplications, a weight matrix can be converted into a set of constrained fine-grained sparse weight matrices. The conversion process may include receiving a request to perform a matrix multiplication operation with a weight matrix, and determining that the weight matrix satisfies a sparsity condition to convert the weight matrix into a set of constrained fine-grained sparse weight matrices. The weight matrix can then be converted into a set of constrained fine-grained sparse weight matrices. Computer instructions can then be generated for an integrated circuit device to perform the requested matrix multiplication operation as a set of sparse matrix multiplication operations using the set of constrained fine-grained sparse weight matrices.
1. An integrated circuit device comprising:
an array of processing elements arranged in rows and columns, wherein each processing element includes multiply-and-add circuitry to multiply an input value with a weight value to generate a multiplication result, and add the multiplication result to a partial sum input to generate a partial sum output, and
wherein the integrated circuit device is operable to:
process a first weight matrix by loading the first weight matrix into the array, and performing a first matrix multiplication operation on the first weight matrix using an entirety of the array; and
process a second weight matrix by loading a compressed weight matrix that removes zero values from the second weight matrix into a section of the array, and performing a second matrix multiplication operation on the second weight matrix using the section of the array,
wherein performing the second matrix multiplication operation using the section of the array allows a part of the array not used by the second matrix multiplication operation to be turned off, or be used to perform a third matrix multiplication operation concurrently with the second matrix multiplication operation.
2. The integrated circuit device of claim 1 , wherein the first weight matrix includes at least one column that does not have any zero weight values.
3. The integrated circuit device of claim 1 , wherein the compressed weight matrix has one or more zero weight values.
4. The integrated circuit device of claim 1 , wherein the compressed weight matrix is stored with sparsity information in a memory of the integrated circuit device, wherein the sparsity information indicates a position of each non-zero weight value in the second weight matrix.
5. An integrated circuit device comprising:
an array of processing elements arranged in rows and columns, wherein each processing element includes multiply-and-add circuitry to multiply an input value with a weight value to generate a multiplication result, and add the multiplication result to a partial sum input to generate a partial sum output, and
wherein the integrated circuit device is operable to process a weight matrix that has a minimum of n number of zero values in each column of the weight matrix, n being one or greater, by:
loading a compressed weight matrix into a section of the array, the compressed weight matrix removing the n number of zero elements from each column of the weight matrix; and
performing a matrix multiplication operation on the weight matrix using the section of the array.
6. The integrated circuit device of claim 5 , wherein the matrix multiplication operation is a first matrix multiplication operation, and the integrated circuit device is operable to concurrently perform a second matrix multiplication operation in the array using a part of the array that is not used by the first matrix multiplication operation.
7. The integrated circuit device of claim 5 , wherein the integrated circuit device is operable to power down a part of the array that is not used by the matrix multiplication operation during the matrix multiplication operation.
8. The integrated circuit device of claim 5 , wherein the compressed weight matrix includes at least one row that contains non-zero weight values from different rows of the weight matrix.
9. The integrated circuit device of claim 5 , wherein the compressed weight matrix includes one or more zero weight values from the weight matrix.
10. The integrated circuit device of claim 5 , wherein the compressed weight matrix is associated with sparsity information indicating a position of each non-zero weight value in the weight matrix.
11. The integrated circuit device of claim 10 , wherein the input value being multiplied with the weight value in a processing element during the matrix multiplication operation is selected based on the sparsity information.
12. The integrated circuit device of claim 10 , wherein weight matrix is stored as the compressed weight matrix in a memory of the integrated circuit device.
13. The integrated circuit device of claim 12 , wherein the compressed weight matrix is stored with the sparsity information in the memory of the integrated circuit device.
14. A method performed by an integrated circuit device having an array of processing elements arranged in rows and columns, the method comprising:
loading a compressed weight matrix into a section of the array, the compressed weight matrix removing n number of zero elements from each column of a weight matrix that has a minimum of n number of zero elements in each column of the weight matrix, n being one or greater;
performing a first matrix multiplication operation on the weight matrix using the section of the array; and
performing a second matrix multiplication operation concurrently with the first matrix multiplication operation using a part of the array that is not used by the first matrix multiplication operation.
15. The method of claim 14 , wherein the compressed weight matrix includes at least one row that contains non-zero weight values from different rows of the weight matrix.
16. The method of claim 14 , wherein the compressed weight matrix includes one or more zero weight values from the weight matrix.
17. The method of claim 14 , wherein the compressed weight matrix is associated with sparsity information indicating a position of each non-zero weight value in the weight matrix.
18. The method of claim 17 , wherein the sparsity information is used to select an input value to multiply with a weight value of the compresses weight matrix during the first matrix multiplication operation.
19. The method of claim 17 , comprising storing the weight matrix as the compressed weight matrix in a memory of the integrated circuit device.
20. The method of claim 19 , comprising storing the compressed weight matrix with the sparsity information in the memory of the integrated circuit device.