Method and apparatus for neural network optimized matrix-matrix multiplication (NNMM)
According to embodiments, a method of neural network optimized matrix-matrix multiplication (NNMM) is performed by at least one processor and includes determining a first matrix of input coefficients, and determining a second matrix of weight coefficients of a neural network that is trained so that the second matrix has a predetermined uniform pattern, the predetermined uniform pattern having at least two of the weight coefficients that have a same value. The method further includes performing multiplication of the determined first matrix and the determined second matrix to determine output coefficients.
1. A method of neural network optimized matrix-matrix multiplication (NNMM) for reducing total computation operations performed by at least one processor, and the method comprising:
determining a first matrix of input coefficients;
determining a second matrix of weight coefficients of a neural network,
wherein the neural network is trained to produce the second matrix having a predetermined uniform pattern,
wherein the predetermined uniform pattern has at least two of the weight coefficients that have a same value;
wherein the second matrix is partitioned into partitions;
swapping partitions of the second matrix such that a number of uniform patterns in the second matrix increases after partitions are swapped and generating the swapped second matrix;
loading the first matrix of input coefficients into a register associated with the at least one processor;
loading the second matrix into the register associated with the at least one processor;
generating a result block by performing multiplication of the first matrix and the swapped second matrix using only a first number of computation operations performed by the at least one processor,
wherein the first number of computation operations performed by the at least one processor are fewer than a second number of computation operations performed by the at least one processor when performing multiplication of the first matrix and the second matrix; and
writing the result block into a main memory associated with the at least one processor.
2. The method of claim 1 , wherein the predetermined uniform pattern has each of the partitions having respective ones of the weight coefficients that have a same value.
3. The method of claim 1 , wherein the predetermined uniform pattern has only one of the partitions having respective ones of the weight coefficients that have a same value.
4. The method of claim 1 , wherein the same value is an absolute value or a non-absolute value, and
the same value is an unquantized floating point value or a quantized integer value.
5. The method of claim 1 , wherein the weight coefficients comprise a four-dimensional (4D) tensor, and
the neural network is trained so that the second matrix has the predetermined uniform pattern, after the 4D tensor is reshaped to a three-dimensional (3D) tensor.
6. The method of claim 1 , further comprising outputting output coefficients based on the result block.
7. An apparatus for neural network optimized matrix-matrix multiplication (NNMM) for reducing total computation operations performed during matrix multiplication, the apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
first determining code configured to cause the at least one processor to determine a first matrix of input coefficients;
second determining code configured to cause the at least one processor to determine a second matrix of weight coefficients of a neural network,
wherein the neural network is trained to produce the second matrix having a predetermined uniform pattern,
wherein the predetermined uniform pattern has at least two of the weight coefficients that have a same value,
wherein the second matrix is partitioned into partitions;
swapping code configured to cause the at least one processor to swap partitions of the second matrix such that a number of uniform patterns in the second matrix increases after partitions are swapped and generating the swapped second matrix;
first loading code configured to cause the at least one processor to load the first matrix of input coefficients into a register associated with the at least one processor;
second loading code configured to cause the at least one processor to load the second matrix into the register associated with the at least one processor;
third generating code configured to cause the at least one processor to generate a result block by performing multiplication of the first matrix and the swapped second matrix using only a first number of computation operations performed by the at least one processor,
wherein the first number of computation operations performed by the at least one processor are fewer than a second number of computation operations performed by the at least one processor when performing multiplication of the first matrix and the second matrix; and
writing code configured to cause the at least one processor to write the result block into a main memory associated with the at least one processor.
8. The apparatus of claim 7 , wherein the predetermined uniform pattern has each of the partitions having respective ones of the weight coefficients that have a same value.
9. The apparatus of claim 7 , wherein the predetermined uniform pattern has only one of the partitions having respective ones of the weight coefficients that have a same value.
10. The apparatus of claim 7 , wherein the same value is an absolute value or a non-absolute value, and
the same value is an unquantized floating point value or a quantized integer value.
11. The apparatus of claim 7 , wherein the weight coefficients comprise a four-dimensional (4D) tensor, and
the neural network is trained so that the second matrix has the predetermined uniform pattern, after the 4D tensor is reshaped to a three-dimensional (3D) tensor.
12. The apparatus of claim 7 , further comprising outputting code configured to cause the at least one processor to output coefficients based on the result block.
13. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor for neural network optimized matrix-matrix multiplication (NNMM) for reducing total computation operations performed during matrix multiplication, cause the at least one processor to:
determine a first matrix of input coefficients;
determine a second matrix of weight coefficients of a neural network,
wherein the neural network is trained to produce the second matrix having a predetermined uniform pattern,
wherein the predetermined uniform pattern has at least two of the weight coefficients that have a same value;
wherein the second matrix is partitioned into partitions;
swap partitions of the second matrix such that a number of uniform patterns in the second matrix increases after partitions are swapped and generating the swapped second matrix;
load the first matrix of input coefficients into a register associated with the at least one processor;
load the second matrix into the register associated with the at least one processor;
generate a result block by performing multiplication of the first matrix and the swapped second matrix using only a first number of computation operations performed by the at least one processor,
wherein the first number of computation operations performed by the at least one processor are fewer than a second number of computation operations performed by the at least one processor when performing multiplication of the first matrix and the second matrix; and
write the result block into a main memory associated with the at least one processor.
14. The non-transitory computer-readable medium of claim 13 , wherein the predetermined uniform pattern has each of the partitions having respective ones of the weight coefficients that have a same value.
15. The non-transitory computer-readable medium of claim 13 , wherein the predetermined uniform pattern has only one of the partitions having respective ones of the weight coefficients that have a same value.
16. The non-transitory computer-readable medium of claim 13 , wherein the same value is an absolute value or a non-absolute value, and
the same value is an unquantized floating point value or a quantized integer value.
17. The non-transitory computer-readable medium of claim 13 , wherein the weight coefficients comprise a four-dimensional (4D) tensor, and
the neural network is trained so that the second matrix has the predetermined uniform pattern, after the 4D tensor is reshaped to a three-dimensional (3D) tensor.