Neural network processor based on application specific synthesis specialization parameters
Neural network processors that have been customized based on application specific synthesis specialization parameters and related methods are described. Certain example neural network processors and methods described in the present disclosure expose several major synthesis specialization parameters that can be used for specializing a microarchitecture instance of a neural network processor to specific neural network models including: (1) aligning the native vector dimension to the parameters of the model to minimize padding and waste during model evaluation, (2) increasing lane widths to drive up intra-row-level parallelism, or (3) increasing matrix multiply tiles to exploit sub-matrix parallelism for large neural network models.
1. A method, implemented by a processor, for synthesizing a neural network processor comprising a pipeline for processing instructions, the pipeline including a matrix vector unit, a first multifunction unit, wherein the first multifunction unit is connected to receive an input from the matrix vector unit, a second multifunction unit, wherein the second multifunction unit is connected to receive an output from the first multifunction unit, wherein the matrix vector unit comprises a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, and wherein each of the plurality of tile engines comprises a plurality of dot product units and wherein each of the plurality of dot product units is configured to receive the matrix elements from a matrix register file, the method comprising:
using the processor analyzing a neural network model corresponding to an application to determine a precision requirement corresponding to each of the matrix elements expressed in a block floating point (BFP) format and corresponding to each of the vector elements expressed in the BFP format;
obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor;
obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements, (2) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and (3) a third synthesis specialization parameter corresponding to a number of the plurality of the engines; wherein each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node, and wherein each of the first native dimension of the each of the matrix elements and the second native dimension of the each of the vector elements is selected to be equal to a block size of the BFP format and wherein the plurality of synthesis specialization parameters further comprises a fourth synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an add-reduction tree; and
using the processor modifying the code; based on at least the first value corresponding to the first synthesis specialization parameter, the second value corresponding to the second synthesis specialization parameter, the third value corresponding to the third synthesis specialization parameter, and a determination regarding inclusion of the add-reduction tree based on the fourth synthesis specialization parameter to generate a modified version of the code and storing a modified version of the code.
2. The method of claim 1 , wherein the plurality of synthesis specialization parameters further comprises a fourth synthesis specialization parameter corresponding to a number of parallel multipliers that can process the matrix elements and the vector elements to produce a partial dot-product.
3. The method of claim 2 , wherein the plurality of synthesis specialization parameters further comprises a fifth synthesis specialization parameter corresponding to a number of independent parallel channels of the plurality of tile engines.
4. The method of claim 3 , wherein the plurality of synthesis specialization parameters further comprises a sixth synthesis specialization parameter corresponding to a number of groups, wherein each of the groups has a group size equal to the number of the plurality of tile engines divided by the number of the independent parallel channels.
5. The method of claim 1 , wherein the at least one hardware node comprises a field programmable gate array (FPGA) including adaptive logic modules, digital signal processors, and random-access memories, and wherein the performance metric corresponds to an area required to implement the adaptive logic modules, the digital signal processors, and the random-access memories as part of the FPGA.
6. The method of claim 1 , wherein the plurality of synthesis specialization parameters further comprises a seventh synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines.
7. The method of claim 1 , wherein the performance metric corresponds to an area required to implement each of the plurality of the engines.
8. A system comprising:
a processor; and
a memory comprising: (1) code representative of at least a portion of at least one hardware node for implementing the neural network processor comprising a pipeline for processing instructions, the pipeline including a matrix vector unit, a first multifunction unit, wherein the first multifunction unit is connected to receive an input from the matrix vector unit, a second multifunction unit, wherein the second multifunction unit is connected to receive an output from the first multifunction unit, wherein the matrix vector unit comprises a plurality of tile engines, wherein each of the plurality of tile engines is configured to process matrix elements and vector elements, and wherein each of the plurality of tile engines comprises a plurality of dot product units and wherein each of the plurality of dot product units is configured to receive the matrix elements from a matrix register file, (2) a synthesis model comprising a plurality of synthesis specialization parameters including: (a) a first synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements and, (b) a second synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, (c) a third synthesis specialization parameter corresponding to a number of the plurality of tile engines, wherein each of a first value corresponding to the first synthesis specialization parameter, a second value corresponding to the second synthesis specialization parameter, and a third value corresponding to the third synthesis specialization parameter is selected to meet or exceed a performance metric associated with the at least one hardware node, and (d) a fourth synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an add-reduction tree, and (3) instructions for synthesizing the neural network processor, the instructions configured to:
using the processor, analyze a neural network model corresponding to an application to determine a precision requirement corresponding to each of the matrix elements expressed in a block floating point (BFP) format and corresponding to each of the vector elements expressed in the BFP format, and wherein each of the first native dimension of the each of the matrix elements and the second native dimension of the each of the vector elements is selected to be equal to a block size of the BFP format,
using the processor modify the code, based on at least the first value corresponding to the first synthesis specialization parameter, the second value corresponding to the second synthesis specialization parameter, the third value corresponding to the third synthesis specialization parameter, and a determination regarding inclusion of the add-reduction tree based on the fourth synthesis specialization parameter, to generate a modified version of the code and store a modified version of the code.
9. The system of claim 8 , wherein the plurality of synthesis specialization parameters further comprises a fourth fifth synthesis specialization parameter corresponding to a number of parallel multipliers that can process the matrix elements and the vector elements to produce a partial dot-product.
10. The system of claim 9 , wherein the plurality of synthesis specialization parameters further comprises a sixth synthesis specialization parameter corresponding to a number of independent parallel channels of the plurality of tile engines.
11. The system of claim 10 , wherein the plurality of synthesis specialization parameters further comprises a seventh synthesis specialization parameter corresponding to a number of groups, wherein each of the groups has a group size equal to the number of the plurality of the engines divided by the number of the independent parallel channels.
12. The system of claim 8 , wherein the at least one hardware node comprises a field programmable gate array (FPGA) including adaptive logic modules, digital signal processors, and random-access memories, and wherein the performance metric corresponds to an area required to implement the adaptive logic modules, the digital signal processors, and the random-access memories as part of the FPGA.
13. The system of claim 8 , wherein the plurality of synthesis specialization parameters further comprises an eighth synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines.
14. A method, implemented by a processor, for synthesizing a neural network processor comprising a pipeline for processing instructions, the pipeline including a matrix vector unit, a first multifunction unit, wherein the first multifunction unit is connected to receive an input from the matrix vector unit, a second multifunction unit, wherein the second multifunction unit is connected to receive an output from the first multifunction unit, wherein the matrix vector unit comprises a plurality of tile engines, wherein each of the plurality of the engines is configured to process matrix elements and vector elements, and wherein each of the plurality of the engines comprises a plurality of dot product units and wherein each of the dot product units is configured to receive the matrix elements from a matrix register file, the method comprising:
using the processor analyzing a neural network model corresponding to an application to determine a precision requirement corresponding to each of the matrix elements expressed in a block floating point (BFP) format and corresponding to each of the vector elements expressed in the BFP format;
obtaining code representative of at least a portion of at least one hardware node for implementing the neural network processor;
obtaining a synthesis model comprising a plurality of synthesis specialization parameters including: (1) a first synthesis specialization parameter corresponding to whether the matrix register file is private to each one of the plurality of tile engines or whether the matrix register file is shared among the plurality of tile engines, (2) a second synthesis specialization parameter corresponding to whether each of the plurality of dot product units comprises an add-reduction tree, (3) a third synthesis specialization parameter corresponding to a first native dimension of the each of the matrix elements, and (4) a fourth synthesis specialization parameter corresponding to a second native dimension of the each of the vector elements, and wherein each of the first native dimension of the each of the matrix elements and the second native dimension of the each of the vector elements is selected to be equal to a block size of the BFP format; and
using the processor modifying the code, based on at least the second synthesis specialization parameter, the third synthesis specialization factor, and the fourth synthesis specialization factor, and storing a modified version of the code.
15. The method of claim 14 , wherein the plurality of synthesis specialization parameters further comprises a fifth synthesis specialization parameter corresponding to a number of parallel multipliers that can process the matrix elements and the vector elements to produce a partial dot-product.
16. The method of claim 15 , wherein the plurality of synthesis specialization parameters further comprises a sixth synthesis specialization parameter corresponding to a number of independent parallel channels of the plurality of tile engines.
17. The method of claim 16 , wherein the plurality of synthesis specialization parameters further comprises a seventh synthesis specialization parameter corresponding to a number of groups, wherein each of the groups has a group size equal to the number of the plurality of tile engines divided by the number of the independent parallel channels of the plurality of tile engines.
18. The method of claim 14 , wherein the at least one hardware node comprises a field programmable gate array (FPGA) including adaptive logic modules, digital signal processors, and random-access memories.