Dot product operations on sparse matrix elements
Embodiments described herein include software, firmware, and hardware logic that provides techniques to perform arithmetic on sparse data via a systolic processing unit. One embodiment provides for data aware sparsity via compressed bitstreams. One embodiment provides for block sparse dot product instructions. One embodiment provides for a depth-wise adapter for a systolic array.
1. A general purpose graphics processor comprising:
a compute cluster including multiple processing resources coupled with a cache memory, a processing resource of the multiple processing resources including a matrix accelerator, the matrix accelerator configured to perform a dot product operation on multiple elements of a sparse first matrix and a second matrix in response to a sparse dot product instruction, wherein the sparse first matrix has a structured sparsity in which elements of the sparse first matrix have been pruned to a predetermined pattern that includes a zero value element and a non-zero value element, elements of the sparse first matrix are compacted, based on the structured sparsity, into a compressed representation that includes the non-zero value element and an indication of the non-zero value element, and the sparse dot product instruction is to cause the matrix accelerator to skip computations associated with input including a zero value element;
wherein the compressed representation is to be stored to the cache memory in a compressed format; and
wherein, in response to the sparse dot product instruction, the processing resource is configured to:
load the compressed representation from the cache memory into a memory within the processing resource;
load the second matrix from the cache memory into the memory within the processing resource;
perform the dot product operation via the matrix accelerator on elements from the compressed representation and selected elements of the second matrix, wherein the selected elements of the second matrix correspond with non-zero values of the sparse first matrix stored within the compressed representation and are selected by the matrix accelerator from the elements of the second matrix stored in the memory within the processing resource, wherein the memory within the processing resource includes a register file and the matrix accelerator is configured to select the selected elements of the second matrix from a vector in the register file based on indications of the non-zero values of the sparse first matrix, the vector including a plurality of elements of the second matrix; and
write output of the dot product operation to the memory within the processing resource.
2. The general purpose graphics processor as in claim 1 , wherein the cache memory is a level two (L2) cache memory.
3. The general purpose graphics processor as in claim 1 , wherein the memory within the processing resource includes a level one (L1) cache memory.
4. The general purpose graphics processor as in claim 1 , wherein the memory within the processing resource additionally includes a shared memory.
5. The general purpose graphics processor as in claim 1 , wherein the matrix accelerator is configured to associate elements from the compressed representation and corresponding selected elements of the second matrix with corresponding channels of input data for processing within the matrix accelerator.
6. The general purpose graphics processor as in claim 1 , wherein the sparse first matrix includes weight data associated with a neural network and the second matrix includes input activation data associated with the neural network.
7. The general purpose graphics processor as in claim 6 , wherein the output of the dot product operation includes output activation data associated with the neural network.
8. The general purpose graphics processor as in claim 7 , wherein the output of the dot product operation is a dense matrix.
9. The general purpose graphics processor as in claim 1 , wherein the matrix accelerator includes a systolic array of processing elements.
10. The general purpose graphics processor as in claim 1 , wherein the dot product operation is an 8-bit integer dot product operation.
11. The general purpose graphics processor as in claim 10 , wherein the sparse first matrix includes 8-bit integer elements.
12. A method comprising:
on a general-purpose graphics processor:
performing a dot product operation on multiple elements of a sparse first matrix and a second matrix in response to a sparse dot product instruction, the dot product operation performed via a compute cluster including multiple processing resources coupled with a cache memory, a processing resource of the multiple processing resources including a matrix accelerator, wherein the sparse first matrix has a structured sparsity in which elements of the sparse first matrix have been pruned to a predetermined pattern that includes a zero value element and a non-zero value element, elements of the sparse first matrix are compacted, based on the structured sparsity, into a compressed representation that includes the non-zero value element and an indication of the non-zero value element, and the sparse dot product instruction is to cause the matrix accelerator to skip computations associated with input including a zero value element;
storing the compressed representation to the cache memory in a compressed format; and
via the processing resource and in response to the sparse dot product instruction:
loading the compressed representation from the cache memory into a memory within the processing resource;
loading the second matrix from the cache memory into the memory within the processing resource;
performing the dot product operation via the matrix accelerator on elements from the compressed representation and selected elements of the second matrix, wherein the selected elements of the second matrix correspond with non-zero values of the sparse first matrix stored within the compressed representation and are selected by the matrix accelerator from the elements of the second matrix stored in the memory within the processing resource, wherein the memory within the processing resource includes a register file and the matrix accelerator is configured to select the selected elements of the second matrix from a vector in the register file based on indications of the non-zero values of the sparse first matrix, the vector including a plurality of elements of the second matrix; and
writing output of the dot product operation to the memory within the processing resource.
13. The method as in claim 12 , wherein the method further comprises compacting the elements of the sparse first matrix into the compressed representation within a memory of the processing resource.
14. The method as in claim 12 , wherein performing a dot product operation includes performing an 8-bit integer dot product operation.
15. The method as in claim 14 , wherein the sparse first matrix includes 8-bit integer elements.
16. A data processing system comprising:
a memory device; and
a general purpose graphics processor comprising a compute cluster including multiple processing resources coupled with a cache memory, a processing resource of the multiple processing resources including a matrix accelerator wherein the matrix accelerator includes a systolic array of processing elements, the matrix accelerator configured to perform a dot product operation on multiple elements of a sparse first matrix and a second matrix in response to a sparse dot product instruction, wherein the sparse first matrix has a structured sparsity in which elements of the sparse first matrix have been pruned to a predetermined pattern that includes a zero value element and a non-zero value element, elements of the sparse first matrix are compacted, based on the structured sparsity, into a compressed representation that includes the non-zero value element and an indication of the non-zero value element, and the sparse dot product instruction is to cause the matrix accelerator to skip computations associated with input including a zero value element;
wherein the compressed representation is to be stored to the cache memory in a compressed format; and
wherein, in response to the sparse dot product instruction, the processing resource is configured to:
load the compressed representation from the cache memory into a memory within the processing resource;
load the second matrix from the cache memory into the memory within the processing resource;
perform the dot product operation via the matrix accelerator on elements from the compressed representation and selected elements of the second matrix, wherein the selected elements of the second matrix correspond with non-zero values of the sparse first matrix stored within the compressed representation and are selected by the matrix accelerator from the elements of the second matrix stored in the memory within the processing resource from the memory within the processing resource, wherein the memory within the processing resource includes a register file and the matrix accelerator is configured to select the selected elements of the second matrix from a vector in the register file based on indications of the non-zero values of the sparse first matrix, the vector including a plurality of elements of the second matrix; and
write output of the dot product operation to the memory within the processing resource.
17. The data processing system as in claim 16 , wherein the cache memory is a level two (L2) cache memory or a level one (L1) cache memory and the memory within the processing resource additionally includes a shared memory.
18. The data processing system as in claim 16 , wherein the sparse first matrix includes weight data associated with a neural network and the second matrix includes input activation data associated with the neural network.
19. The data processing system as in claim 16 , wherein the dot product operation is an 8-bit integer dot product operation and the sparse first matrix includes 8-bit integer elements.
20. A general purpose graphics processor comprising:
a compute cluster including multiple processing resources coupled with a cache memory, a processing resource of the multiple processing resources including a matrix accelerator, wherein the matrix accelerator includes a systolic array of processing elements and the matrix accelerator is configured to:
perform a dot product operation on multiple elements of a sparse first matrix and a second matrix in response to a sparse dot product instruction, wherein the sparse dot product instruction is to cause the matrix accelerator to skip computations associated with input including a zero value element, wherein the sparse first matrix has a structured sparsity in which elements of the sparse first matrix have been pruned to a predetermined pattern that includes a zero value element and a non-zero value element, elements of the sparse first matrix are compacted, based on the structured sparsity, into a compressed representation that includes the non-zero value element and an indication of a location of the non-zero value element within a bitstream of packed data elements,
wherein the dot product operation is performed on elements from the compressed representation and selected elements of the second matrix in response to the sparse dot product instruction, the selected elements of the second matrix correspond with non-zero values of the sparse first matrix stored within the compressed representation, and are selected by the matrix accelerator from the elements of the second matrix stored in a memory within the processing resource, and
wherein the memory within the processing resource includes a register file and the matrix accelerator is configured to select the selected elements of the second matrix from a vector in the register file based on indications of the locations of the non-zero value elements within the bitstream of packed data elements, the vector including a plurality of elements of the second matrix; and
write output of the dot product operation to the memory within the processing resource.
21. The general purpose graphics processor as in claim 20 , wherein to perform the dot product operation includes to perform an 8-bit integer dot product operation.
22. The general purpose graphics processor as in claim 21 , wherein the sparse first matrix includes 8-bit integer elements.