Systems and methods for generation of sparse-dense matrix multiplication code for neural network and other applications
Generating computer code to perform a matrix-multiply operation, for use in neural network compute or other applications may include determining parameters R r (number of rows for a plurality of partitioned matrices A′), Rc (number of registers used for each column of a target matrix C′), and R L (number of processor registers used as a source for targets for vector broadcast instructions. For each row r in A′, having R r rows, operations may be performed such that for each column i, if the element at row r and column i is non-zero: a vector broadcast instruction may be generate to a register for the element at row r and column i; and a set of fused multiply-add (FMA) instructions may be generated having as parameters the register, a set of registers for a target matrix, and a memory location for input data.
1 . A method comprising, prior to an inference phase of a neural network:
generating computer instruction code for execution in relation to a matrix-multiply operation of the neural network, wherein generating the computer instruction code involves:
for each row r and each column i in a sparse neural network parameter matrix associated with the neural network:
analyzing an element at row r and column i to determine whether the element is a zero element or a non-zero element; and
if the element at row r and column i is a non-zero element:
generating, in the computer instruction code, a vector broadcast instruction to a register for the element at row r and column i; and
generating, in the computer instruction code, a set of fused multiply-add (FMA) instructions having as parameters the register, a set of registers for a target matrix, and a memory location for input data; or
if the element at row r and column i is a zero element, omitting corresponding instructions for the element from the computer instruction code; and
compiling the computer instruction code into compiled code, wherein the compiled code is configured to be subsequently executed during the inference phase of the neural network to perform the matrix-multiply operation using the vector broadcast instruction and set of FMA instructions generated for each non-zero element.
2 . The method of claim 1 , wherein generating a vector broadcast instruction to a register comprises:
selecting a next available register from a circular queue of registers for use as the register.
3 . The method of claim 1 , further comprising incorporating, into the computer instruction code, an additional set of instructions to combine more than one produced target matrix into a final matrix.
4 . The method of claim 1 , further comprising determining R r , R c and R L , where R r is the number of rows for the sparse neural network parameter matrix, R c is the number of registers used for each row of the target matrix, and R L is the number of registers in a queue of registers used as a source for the register for which the vector broadcast instruction is generated.
5 . The method of claim 1 , wherein the input data is represented as an input data matrix, the method further comprising determining R r and R c , where R r is the number of rows for the target matrix, and Rc is the number of registers used for each row of the input data matrix.
6 . A method comprising, prior to an inference phase of a neural network:
generating computer instruction code for execution in relation to a matrix-multiply operation of the neural network, wherein generating the computing instruction code involves:
for each row r and each column i in a first matrix associated with the neural network:
analyzing an element at row r and column i to determine whether the element is a zero element or a non-zero element; and
if the element at row r and column i is a non-zero element:
selecting a next available register from a circular queue of registers;
generating, in the computer instruction code, a vector broadcast instruction to the selected register for the element at row r and column i; and
generating, in the computer instruction code, a set of fused multiply-add (FMA) instructions having as parameters the selected register, a set of registers for a target matrix, and a memory location for input data; or
if the element at row r and column i is a zero element, omitting corresponding instructions for the element from the computer instruction code; and
compiling the computer instruction code into compiled code, wherein the compiled code is configured to be subsequently executed during the inference phase of the neural network to perform the matrix-multiply operation using the vector broadcast instruction and set of FMA instructions generated for each non-zero element.
7 . The method of claim 6 , further comprising incorporating, into the computer instruction code, an additional set of instructions to combine more than one produced target matrix into a final matrix.
8 . The method of claim 6 , further comprising determining R r , R c and R L , where R r is the number of rows for the first matrix, R c is the number of registers used for each row of the target matrix, and R L is the number of registers in the circular queue of registers used as a source for the register for which the vector broadcast instruction is generated.
9 . The method of claim 6 , wherein the input data is represented as an input data matrix, the method further comprising determining R r and R c , where R r is the number of rows for the target matrix, and Rc is the number of registers used for each row of the input data matrix.
10 . The method of claim 6 , where the first matrix represents neural network parameters.
11 . A system comprising:
a processor; and
a memory storing instructions that are executable by the processor for causing the processor to perform operations including, prior to an inference phase of a neural network:
generating computer instruction code for execution in relation to a matrix-multiply operation of the neural network, wherein generating the computer instruction code involves:
for each row r and each column i in a sparse neural network parameter matrix associated with the neural network:
analyzing an element at row r and column i to determine whether the element is a zero element or a non-zero element; and
if the element at row r and column i is a non-zero element:
generating, in the computer instruction code, a vector broadcast instruction to a register for the element at row r and column i; and
generating, in the computer instruction code, a set of fused multiply-add (FMA) instructions having as parameters the register, a set of registers for a target matrix, and a memory location for input data; or
if the element at row r and column i is a zero element, omitting corresponding instructions for the element from the computer instruction code; and
compiling the computer instruction code into compiled code, wherein the compiled code is configured to be subsequently executed during the inference phase of the neural network to perform the matrix-multiply operation using the vector broadcast instruction and set of FMA instructions generated for each non-zero element.
12 . The system of claim 11 , wherein generating a vector broadcast instruction to a register comprises:
selecting a next available register from a circular queue of registers for use as the register.
13 . The system of claim 11 , wherein the operations further comprise:
incorporating, into the computer instruction code, an additional set of instructions to combine more than one produced target matrix into a final matrix.
14 . The system of claim 11 , wherein the operations further comprise:
determining R r , R c and R L , where R r is the number of rows for the sparse neural network parameter matrix, R c is the number of registers used for each row of the target matrix, and R L is the number of registers in a queue of registers used as a source for the register for which the vector broadcast instruction is generated.
15 . The system of claim 11 , wherein the input data is represented as an input data matrix, and wherein the operations further comprise determining R r and R c , where R r is the number of rows for the target matrix, and Rc is the number of registers used for each row of the input data matrix.
16 . A system comprising:
a processor; and
a memory storing instructions that are executable by the processor for causing the processor to perform operations including, prior to an inference phase of a neural network:
generating computer instruction code for execution in relation to a matrix-multiply operation of the neural network, wherein generating the computer instruction code involves:
for each row r and each column i in a first matrix associated with the neural network:
analyzing an element at row r and column i to determine whether the element is a zero element or a non-zero element; and
if the element at row r and column i is a non-zero element:
selecting a next available register from a circular queue of registers;
generating, in the computer instruction code, a vector broadcast instruction to the selected register for the element at row r and column i; and
generating, in the computer instruction code, a set of fused multiply-add (FMA) instructions having as parameters the selected register, a set of registers for a target matrix, and a memory location for input data; or
if the element at row r and column i is a zero element, omitting corresponding instructions for the element from the computer instruction code; and
compiling the computer instruction code into compiled code, wherein the compiled code is configured to be subsequently executed during the inference phase of the neural network to perform the matrix-multiply operation using the vector broadcast instruction and set of FMA instructions generated for each non-zero element.
17 . The system of claim 16 , wherein the operations further comprise:
incorporating, into the computer instruction code, an additional set of instructions to combine more than one produced target matrix into a final matrix.