Compiler for optimizing filter sparsity for neural network implementation configuration
Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). In some embodiments, the compiler determines whether sparsity requirements of channels implemented on individual cores are met on each core. If the sparsity requirement is not met, the compiler, in some embodiments, determines whether the channels of the filter can be rearranged to meet the sparsity requirements on each core and, based on the determination, either rearranges the filter channels or implements a solution to non-sparsity.
1. A method for generating neural network program instructions for a neural network inference circuit (NNIC) that comprises a plurality of cores for computing dot products between weight values and input values to execute a neural network comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node, the method comprising:
receiving configuration data, for executing the neural network, comprising a first assignment of sets of channels of a particular filter to a set of cores of the NNIC, each core in the set of cores (i) being assigned a different plurality of the channels and (ii) comprising a plurality of partial dot product computation circuits (PDPCCs) for computing dot products, wherein for a particular computation node that uses the particular filter, addition circuitry of the NNIC adds together outputs of PDPCCs from each core of the set of cores to which the sets of channels of the particular filter are assigned;
when a particular set of channels of the particular filter does not have enough weight values equal to zero for a first core to which the particular set of channels is assigned to compute partial dot products for the computation nodes that use the particular filter, identifying a second assignment of sets of channels of the particular filter to the set of cores such that each set of channels assigned to a respective core has enough weight values equal to zero for the respective core to compute partial dot products for the computation nodes that use the particular filter; and
based on the identification, generating a set of program instructions for the neural network inference circuit to use to execute the neural network using the second identified assignment of filter channels.
2. The method of claim 1 , wherein the first assignment of sets of channels of the particular filter to the set of cores assigns each set of channels of the first assignment to one PDPCC of each core of the set of cores.
3. The method of claim 2 , wherein the second assignment of sets of channels of the particular filter to the set of cores assigns each set of channels of the second assignment to the same one PDPCC of each core of the set of cores.
4. The method of claim 2 , wherein the one PDPCC of each core is addressed by a common index in its respective core.
5. The method of claim 4 , wherein a first set of channels in the first assignment that is assigned to a particular core is identical to a second set of channels in the second assignment that is assigned to the particular core.
6. The method of claim 1 , wherein each PDPCC in the set of PDPCCs comprises a plurality of multiplexers for a set of input values associated with the set of channels assigned to the PDPCC, the multiplexers selecting input values associated with non-zero weight values for the set of channels assigned to the PDPCC for calculating a product of the input values and the non-zero weight values.
7. The method of claim 6 , wherein the method further comprises determining (1) that the particular set of channels does not have enough zero weight values and (2) that the sets of channels of the filter cannot be rearranged into a same number of sets of channels such that each set of channels in the rearranged sets of channels has enough zero weight values,
wherein the second assignment of sets of channels of the filter to different cores comprises dividing at least one of the sets of channels between two PDPCCs in a same core.
8. The method of claim 6 , wherein the number of multiplexers in each PDPCC is less than the number of weight values in the set of channels assigned to the PDPCC.
9. The method of claim 1 , wherein the first assignment of sets of channels requires a single clock cycle in which a set of computations is completed for a particular computation node that uses the particular filter, wherein the second assignment requires time-multiplexing in which the set of computations is completed for the particular computation node using two clock cycles.
10. A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for generating neural network program instructions for a neural network inference circuit (NNIC) that comprises a plurality of cores for computing dot products between weight values and input values to execute a neural network comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node, the program comprising sets of instructions for:
receiving configuration data, for executing the neural network, comprising a first assignment of sets of channels of a particular filter to a set of cores of the NNIC, each core in the set of cores (i) being assigned a different plurality of the channels and (ii) comprising a plurality of partial dot product computation circuits (PDPCCs) for computing dot products, wherein for a particular computation node that uses the particular filter, addition circuitry of the NNIC adds together outputs of PDPCCs from each core of the set of cores to which the sets of channels of the particular filter are assigned;
when a particular set of channels of the particular filter does not have enough weight values equal to zero for a first core to which the particular set of channels is assigned to compute partial dot products for the computation nodes that use the particular filter, identifying a second assignment of sets of channels of the particular filter to the set of cores such that each set of channels assigned to a respective core has enough weight values equal to zero for the respective core to compute partial dot products for the computation nodes that use the particular filter; and
based on the identification, generating a set of program instructions for the neural network inference circuit to use to execute the neural network using the second identified assignment of filter channels.
11. The non-transitory machine readable medium of claim 10 , wherein the first assignment of sets of channels of the particular filter to the set of cores assigns each set of channels of the first assignment to one PDPCC of each core of the set of cores.
12. The non-transitory machine readable medium of claim 11 , wherein the second assignment of sets of channels of the particular filter to the set of cores assigns each set of channels of the second assignment to the same one PDPCC of each core of the set of cores.
13. The non-transitory machine readable medium of claim 11 , wherein the one PDPCC of each core is addressed by a common index in its respective core.
14. The non-transitory machine readable medium of claim 13 , wherein a first set of channels in the first assignment that is assigned to a particular core is identical to a second set of channels in the second assignment that is assigned to the particular core.
15. The non-transitory machine readable medium of claim 10 , wherein each PDPCC in the set of PDPCCs comprises a plurality of multiplexers for a set of input values associated with the set of channels assigned to the PDPCC, the multiplexers selecting input values associated with non-zero weight values for the set of channels assigned to the PDPCC for calculating a product of the input values and the non-zero weight values.
16. The non-transitory machine readable medium of claim 15 further comprising sets of instructions for determining (1) that the particular set of channels does not have enough zero weight values and (2) that the sets of channels of the filter cannot be rearranged into a same number of sets of channels such that each set of channels in the rearranged sets of channels has enough zero weight values,
wherein the second assignment of sets of channels of the filter to different cores comprises dividing at least one of the sets of channels between two PDPCCs in a same core.
17. The non-transitory machine readable medium of claim 15 , wherein the number of multiplexers in each PDPCC is less than the number of weight values in the set of channels assigned to the PDPCC.
18. The non-transitory machine readable medium of claim 10 , wherein the first assignment of sets of channels requires a single clock cycle in which a set of computations is completed for a particular computation node that uses the particular filter, wherein the second assignment requires time-multiplexing in which the set of computations is completed for the particular computation node using two clock cycles.