SYSTEM AND METHOD OF NEURAL NETWORK PROCESSING REDUCING INSTRUCTION USAGE
A system and method for executing or training a neural network (NN) may, using a computer processor, for a matrix A, for each row in A, for each unique value z appearing in one or more locations in the row in A: summing the set of rows in a matrix B where the set of rows in matrix B correspond to the indices of z in the row in A, the summing producing a vector, multiplying the vector by the unique value z to produce a product vector; and adding the product vector to a row in an output matrix C which corresponds to the row in A.
1 . A method of executing a neural network (NN), the method comprising, using a computer processor:
for a matrix A, for each row in A, for each unique value z appearing in one or more locations in the row in A;
summing the set of rows in a matrix B where the set of rows in matrix B correspond to the indices of z in the row in A, the summing producing a vector;
multiplying the vector by the unique value z to produce a product vector; and
adding the product vector to a row in an output matrix C which corresponds to the row in A.
2 . The method of claim 1 , wherein each value in A is quantized such that each value in A is represented by three or fewer bits; and wherein each value in B is not quantized, such that each value in B is represented by eight or more bits.
3 . The method of claim 1 , comprising performing inference on the NN by accepting an input to the NN and producing an output from the NN.
4 . The method of claim 1 , wherein the summing uses add CPU instructions and the multiplication uses multiply CPU instructions.
5 . The method of claim 1 , wherein the summing uses vector add instructions and the multiplication uses vector multiply instructions.
6 . The method of claim 1 , comprising producing code for the summing and multiplying based on an input of the matrix A.
7 . The method of claim 1 , wherein matrix A stores weights or a kernel.
8 . The method of claim 1 , wherein summing the set of rows in the matrix B comprises partitioning each of the set of rows in the matrix B such that the summing occurs for each partition.
9 . A system for executing a neural network (NN), the system comprising:
a memory;
a computer processor to:
for a matrix A, for each row in A, for each unique value z appearing in one or more locations in the row in A;
sum the set of rows in a matrix B where the set of rows in matrix B correspond to the indices of z in the row in A, the summing producing a vector;
multiply the vector by the unique value z to produce a product vector; and
add the product vector to a row in an output matrix C which corresponds to the row in A.
10 . The system of claim 9 , wherein each value in A is quantized such that each value in A is represented by three or fewer bits; and wherein each value in B is not quantized, such that each value in B is represented by eight or more bits.
11 . The system of claim 9 , wherein the computer processor is to perform inference on the NN by accepting an input to the NN and producing an output from the NN.
12 . The system of claim 9 , wherein the summing uses add CPU instructions and the multiplication uses multiply CPU instructions.
13 . The system of claim 9 , wherein the summing uses vector add instructions and the multiplication uses vector multiply instructions.
14 . The system of claim 9 , wherein the computer processor is to produce code for the summing and multiplying based on an input of the matrix A.
15 . The system of claim 9 , wherein matrix A stores weights or a kernel.
16 . The system of claim 9 , wherein summing the set of rows in the matrix B comprises partitioning each of the set of rows in the matrix B such that the summing occurs for each partition.
17 . A method of executing a neural network (NN), the method comprising:
summing activation values in an activation tensor which correspond to one unique value in a weight tensor; and
multiplying the resulting sum by the unique value.
18 . The method of claim 17 , wherein the summing uses vector add instructions and the multiplying uses vector multiply instructions.
19 . The method of claim 17 , comprising performing inference on the NN by accepting an input to the NN and producing an output from the NN.
20 . The method of claim 17 , wherein the summing uses add CPU instructions and the multiply uses multiply CPU instructions.