Systems and methods for activation sparse and kernel sparse general matrix multiplication in neural networks
A system and method for performing multiplication for a neural network, e.g. for data of one or more layers in a neural network, may include loading a portion of a compressed version of a sparse input matrix into a cache memory; uncompressing a subset of the data in the portion of the compressed version of the sparse input matrix; and multiplying a sparse kernel matrix by the subset of the data using a set of instructions which are themselves created based on the sparse kernel matrix.
1 . A method of executing a layer of a neural network (NN), the method comprising:
loading, by a processor and from a memory external to the processor, a portion of a compressed version of an input matrix into a cache, the cache being local to the processor, wherein the input matrix comprises data output from a previous layer of the NN or input data to a first layer of the NN, wherein the previous layer is prior to the layer of the NN to be executed;
after loading the portion of the compressed version of the input matrix into the cache, generating, by the processor, an uncompressed portion of the input matrix by uncompressing the loaded portion of the compressed version of the input matrix, to avoid subsequently using instructions that perform multiplication on the compressed version of the input matrix when executing the layer of the NN, wherein the uncompressed portion is stored in the cache that is local to the processor;
after the uncompressed portion is stored in the cache, executing, by the processor, the layer of the NN by using the uncompressed portion of the input matrix stored in the cache, wherein the executing involves multiplying a kernel matrix by the uncompressed portion of the input matrix using a set of instructions, wherein the set of instructions were pre-generated prior to the execution of the layer of the NN based on the kernel matrix and relate to non-zero elements of the kernel matrix; and
outputting, by the processor, an output generated by the NN based at least in part on the execution of the layer of the NN.
2 . The method of claim 1 , wherein the compressed version of the input matrix is compressed using a compressed sparse column (CSC) format.
3 . The method of claim 1 , wherein the set of instructions do not include instructions relating to zero elements of the kernel matrix.
4 . The method of claim 1 , comprising repeating the loading, uncompressing and multiplying operations until all of the input matrix has been multiplied.
5 . A system for executing a layer of a neural network (NN), the system comprising:
a memory; and
a processor comprising a cache local to the processor, wherein the memory is external to the processor, and wherein the processor is configured to:
load, from the memory external to the processor, a portion of a compressed version of an input matrix into the cache local to the processor, wherein the input matrix comprises data output from a previous layer of the NN or input data to a first layer of the NN, wherein the previous layer is prior to the layer of the NN to be executed;
after loading the portion of the compressed version of the input matrix into the cache, generate an uncompressed portion of the input matrix by uncompressing the loaded portion of the compressed version of the input matrix, to avoid subsequently using instructions that perform multiplication on the compressed version of the input matrix when executing the layer of the NN, wherein the uncompressed portion is stored in the cache that is local to the processor;
after the uncompressed portion is stored in the cache, execute the layer of the NN by using the uncompressed portion of the input matrix stored in the cache, wherein the executing involves multiplying a kernel matrix by the uncompressed portion of the input matrix using a set of instructions, wherein the set of instructions were pre-generated prior to the execution of the layer of the NN based on the kernel matrix and relate to non-zero elements of the kernel matrix; and
outputting an output generated by the NN based at least in part on the execution of the layer of the NN.
6 . The system of claim 5 , wherein the compressed version of the input matrix is compressed using a compressed sparse column (CSC) format.
7 . The system of claim 5 , wherein the set of instructions do not include instructions relating to zero elements of the kernel matrix.
8 . The system of claim 5 , wherein the processor is configured to repeat the loading, uncompressing and multiplying operations until all of the input matrix has been multiplied.
9 . A method of executing a layer of a neural network (NN), the method comprising:
loading, by a processor and from a memory external to the processor, a compressed input matrix into a cache, the cache being local to the processor, wherein the compressed input matrix represents an output from a previous layer of the NN or input data to a first layer of the NN, wherein the previous layer is prior to the layer of the NN to be executed;
generating, by the processor, uncompressed data by uncompressing data of the compressed input matrix, to avoid subsequently using instructions that perform multiplication on the compressed input matrix when executing the layer of the NN, wherein the uncompressed data is stored in the cache that is local to the processor;
loading, by the processor, a set of instructions into an instruction cache, wherein the set of instructions relate to non-zero elements of a kernel matrix;
after generating the uncompressed data and loading the set of instructions into the instruction cache, executing, by the processor, the layer of the NN, wherein the executing involves multiplying the kernel matrix by the uncompressed data using the set of instructions; and
outputting, by the processor, an output generated by the NN based at least in part on the execution of the layer of the NN.
10 . The method of claim 9 , wherein the compressed input matrix is compressed using a compressed sparse column (CSC) format.
11 . The method of claim 9 , wherein the set of instructions do not include instructions relating to zero elements of the kernel matrix.
12 . The method of claim 9 , comprising repeating the uncompressing and multiplying operations until all subblocks of the input matrix have been multiplied.
13 . The method of claim 9 , wherein the uncompressing and multiplying operations are performed over a series of iterations, the method comprising, prior to the first iteration, uncompressing a buffer of compressed input matrix data.