System and method of executing neural networks using compression and tensor columns
A method and system may operate a neural network (NN), e.g. during inference or training, by executing a first tensor column comprising task instruction code representing at least one computation spanning a number of layers of the NN, the execution producing an output, and compressing that output. In order to execute a next tensor column, the output may be uncompressed to produce uncompressed output; and the second tensor column may be executed, the second tensor column including task instruction code representing at least one computation spanning a number of layers of the NN. The second tensor column may take as input the uncompressed output.
1 . A method performed by at least one processor, the method comprising:
receiving information indicating a size of a cache memory of a processor core on which at least a portion of a neural network (NN) is to be executed;
partitioning the NN into a plurality of tensor columns based on the size of the cache memory of the processor core, wherein each tensor column of the plurality of tensor columns is adapted to fit within the cache memory of the processor core, and wherein each tensor column of the plurality of tensor columns includes:
a reference to a set of input data on which to perform at least one computation;
a set of task instruction code representing the at least one computation; and
an allocation of output memory space within the cache memory of the processor core;
executing a first tensor column of the plurality of tensor columns, the first tensor column comprising a first set of task instruction code representing at least one computation and spanning only a portion of a layer of each of a first plurality of layers of the NN, the execution producing an output;
compressing the output;
uncompressing the compressed output to produce uncompressed output; and
executing a second tensor column of the plurality of tensor columns, the second tensor column comprising a second set of task instruction code representing at least one computation and spanning only a portion of a layer of each of a second plurality of layers of the NN, the second tensor column taking as input the uncompressed output.
2 . The method of claim 1 , wherein executing the first tensor column comprises multiplying a kernel matrix by an input matrix.
3 . The method of claim 1 , wherein at least one tensor column of the plurality of tensor columns includes as a boundary a layer including an activation function increasing activation sparsity.
4 . The method of claim 1 , wherein the compression takes place at a boundary layer of the first tensor column.
5 . The method of claim 1 , wherein the uncompressed output that the second tensor column takes as input comprises less than the entirety of the output for a layer of the NN.
6 . The method of claim 1 , further comprising storing the compressed output.
7 . The method of claim 1 , wherein the first plurality of layers is the same as the second plurality of layers.
8 . A system comprising:
a memory; and
a processor, configured to:
receive information indicating a size of a cache memory of a processor core on which at least a portion of a neural network (NN) is to be executed;
partition the NN into a plurality of tensor columns based on the size of the cache memory of the processor core, wherein each tensor column of the plurality of tensor columns is adapted to fit within the cache memory of the processor core, and wherein each tensor column of the plurality of tensor columns includes:
a reference to a set of input data on which to perform at least one computation;
a set of task instruction code representing the at least one computation; and
an allocation of output memory space within the cache memory of the processor core;
execute a first tensor column of the plurality of tensor columns, the first tensor column comprising a first set of task instruction code representing at least one computation and spanning only a portion of a layer of each of a first plurality of layers of the NN, the execution producing an output;
compress the output;
uncompress the compressed output to produce uncompressed output; and
execute a second tensor column of the plurality of tensor columns, the second tensor column comprising a second set of task instruction code representing at least one computation and spanning only a portion of a layer of each of a second plurality of layers of the NN, the second tensor column taking as input the uncompressed output.
9 . The system of claim 8 , wherein executing the second tensor column comprises multiplying a kernel matrix by an input matrix.
10 . The system of claim 8 , wherein at least one tensor column of the plurality of tensor columns includes as a boundary a layer including an activation function increasing activation sparsity.
11 . The system of claim 8 , wherein the compression takes place at a boundary layer of the first tensor column.
12 . The system of claim 8 , wherein the uncompressed output that the second tensor column takes as input comprises less than the entirety of the output for a layer of the NN.
13 . The system of claim 8 , wherein the processor is configured to store the compressed output.
14 . The system of claim 8 , wherein the first plurality of layers is different from the second plurality of layers.
15 . A method performed by at least one processor, the method comprising:
receiving information indicating a size of a cache memory of a processor core on which at least a portion of a neural network (NN) is to be executed;
partitioning the NN into a plurality of pyramid tasks based on the size of the cache memory of the processor core, wherein each pyramid task of the plurality of pyramid tasks is adapted to fit within the cache memory of the processor core, and wherein each pyramid task of the plurality of pyramid tasks includes:
a reference to a set of input data on which to perform at least one computation;
a set of task instruction code representing the at least one computation; and
an allocation of output memory space within the cache memory of the processor core;
executing a first pyramid task of the plurality of pyramid tasks, the first pyramid task being executed across only a portion of a layer of each of a first plurality of layers of the NN to produce an output;
compressing the output and storing the compressed output;
retrieving and uncompressing the compressed output to produce uncompressed output; and
using the uncompressed output, executing a second pyramid task of the plurality of pyramid tasks, the second pyramid tasks being executed across only a portion of a layer of each of a second plurality of layers of the NN.
16 . The method of claim 15 , wherein executing at least one pyramid task of the plurality of pyramid tasks comprises multiplying a kernel matrix by an input matrix.
17 . The method of claim 15 , wherein at least one pyramid task of the plurality of pyramid tasks includes as a boundary a layer including an activation function increasing activation sparsity.
18 . The method of claim 15 , wherein the compression takes place at a boundary layer of the first pyramid task.
19 . The method of claim 15 , wherein the uncompressed output that the second pyramid task takes as input consists of less than the entirety of the output for a layer of the NN.
20 . The method of claim 15 , wherein the first plurality of layers is the same as the second plurality of layers.
21 . A method performed by at least one processor, the method comprising:
receiving information indicating a size of a cache memory of a processor core on which at least a portion of a neural network (NN) is to be executed;
partitioning the NN into a plurality of tensor columns based on the size of the cache memory of the processor core, wherein each tensor column of the plurality of tensor columns is adapted to fit within the cache memory of the processor core, and wherein each tensor column of the plurality of tensor columns includes:
a reference to a set of input data on which to perform at least one computation;
a set of task instruction code representing the at least one computation; and
an allocation of output memory space within the cache memory of the processor core; and
executing a first tensor column of the plurality of tensor columns, the first tensor column comprising a first set of task instruction code representing at least one computation and spanning only a portion of a layer of each of a first plurality of layers of the NN, to thereby produce a set of gradients;
compressing the set of gradients;
uncompressing the compressed set of gradients to produce uncompressed gradients; and
performing backpropagation, using the uncompressed gradients, on a second tensor column of the plurality of tensor columns, the second tensor column comprising a second set of task instruction code representing at least one computation and spanning only a portion of a layer of each of a second plurality of layers of the NN.
22 . The method of claim 21 , wherein executing at least one tensor column of the plurality of tensor columns comprises multiplying a kernel matrix by an input matrix.
23 . The method of claim 21 , wherein at least one tensor column of the plurality of tensor columns includes as a boundary a layer including a pooling layer.