Systems and methods for generation of sparse code for convolutional neural networks
A system and method may generate code to be used when executing neural networks (NNs), for example convolutional neural networks (CNNs) which may include one or more convolutional layers. For at least one convolutional layer, for each non-zero element in a kernel tensor or matrix associated with the convolutional layer, instructions may be generated or issued. For example, for each non-zero element, a vector broadcast instruction may be generated, and a fused multiply-add (FMA) instruction may be generated, having as parameters a register representing a portion of the output for the convolutional layer, a register storing input data for the convolutional layer, and a register or reference to memory storing the non-zero element. The software or code produced may be executed during convolutional operations, for example as part of a larger application such as a NN inference application.
1. A method of generating computer instruction code for executing a convolutional neural network (CNN) comprising one or more convolutional layers, the method comprising:
for at least one convolutional layer:
performing within a set of nested loops, an inner loop iterating over 1 to OC, OC being a channel dimension of a produced output tensor, and loops outside the inner loop iterating over dimensions of an input and a kernel tensor associated with the convolutional layer:
for each non-zero element in the kernel tensor generating a fused multiply-add (FMA) instruction having as parameters a portion of the output tensor, input data for the convolutional layer, and the non-zero element.
2. The method of claim 1 wherein:
the portion of the output for the convolutional layer is represented as an FMA parameter as a register;
the input data for the convolutional layer is represented as an FMA parameter as a register; and
the non-zero element is represented as an FMA parameter as a register or reference to a memory location.
3. The method of claim 1 , comprising, for each non-zero element in the kernel tensor generating a vector broadcast instruction to a register.
4. The method of claim 1 , comprising generating a set of instructions to combine more than one produced output tensor into a final output tensor.
5. The method of claim 1 , comprising determining parameters describing the dimensions of a produced output tensor, at least one parameter dependent on the number of registers in a target architecture executing the computer instruction code.
6. The method of claim 1 , comprising creating a plurality of code sections for the at least one convolutional layer, such that during execution of the CNN, at least a first code section is executed in conjunction with code executing at least a portion of a lower CNN layer before the execution of a second code section.
7. A system for generating computer instruction code for executing a convolutional neural network (CNN) comprising one or more convolutional layers, the system comprising:
a memory and;
a processor configured to:
for at least one convolutional layer:
perform within a set of nested loops, an inner loop iterating over 1 to OC, OC being a channel dimension of a produced output tensor, and loops outside the inner loop iterating over dimensions of an input and a kernel tensor associated with the convolutional layer:
for each non-zero element in the kernel tensor associated with the generate a fused multiply-add (FMA) instruction having as parameters a portion of the output tensor, input data for the convolutional layer, and the non-zero element.
8. The system of claim 7 wherein:
the portion of the output for the convolutional layer is represented as an FMA parameter as a register;
the input data for the convolutional layer is represented as an FMA parameter as a register; and
the non-zero element is represented as an FMA parameter as a register or reference to a memory location.
9. The system of claim 7 , wherein the processor is configured to, for each non-zero element in the kernel tensor generate a vector broadcast instruction to a register.
10. The system of claim 7 , wherein the processor is configured to generate a set of instructions to combine more than one produced output tensor into a final output tensor.
11. The system of claim 7 , wherein the processor is configured to determine parameters describing the dimensions of a produced output tensor, at least one parameter dependent on the number of registers in a target architecture executing the computer instruction code.
12. The system of claim 7 , wherein the processor is configured to create a plurality of code sections for the at least one convolutional layer, such that during execution of the CNN, at least a first code section is executed in conjunction with code executing at least a portion of a lower CNN layer before the execution of a second code section.
13. A method for executing a convolutional neural network (CNN) comprising one or more convolutional layers, the method comprising:
for at least one convolutional layer of the CNN:
executing a series of fused multiply-add (FMA) instructions, each FMA instruction taking as input a non-zero element of a kernel tensor associated with the convolutional layer, the series of FMA instructions associated with a data store holding only non-zero elements of the kernel tensor;
wherein the series of FMA instructions are generated by performing instruction generation operations within a set of nested loops, an inner loop iterating over 1 to OC, OC being a channel dimension of an output tensor produced by the series of FMA instructions, and loops outside the inner loop iterating over dimensions of the input and the kernel tensor.
14. The method of claim 13 , comprising executing a vector broadcast instruction to a register, the register input to an FMA instruction.
15. The method of claim 13 , comprising executing a set of instructions to combine more than one produced output tensor into a final output tensor.
16. The method of claim 13 , comprising executing a plurality of code sections for the at least one convolutional layer, such that during execution of the CNN, at least a first code section is executed in conjunction with code executing at least a portion of a lower CNN layer before the execution of a second code section.
17. A method of compiling computer code for executing a convolutional neural network (CNN) layer, the method comprising:
iterating across a portion of a kernel tensor to access each element of the kernel tensor, the iteration using a set of nested loops, an inner loop iterating over 1 to OC, OC being a channel dimension of an output tensor produced by the layer, and loops outside the inner loop iterating over dimensions of an input and the kernel tensor; and
if an accessed element is not zero generating a multiply-accumulate instruction taking as input the accessed element.
18. The method of claim 17 , wherein the multiply-accumulate instruction takes, as input, input data for the convolutional layer.