Splitting neural network filters for implementation by neural network inference circuit
Some embodiments provide a method for compiling a neural network (NN) program for an NN inference circuit (NNIC) that includes multiple partial dot product computation circuits (PDPCCs) for computing dot products between weight values and input values. The method receives an NN definition with multiple nodes. The method assigns a group of filters to specific PDPCCs. Each filter is assigned to a different set of the PDPCCs. When a filter does not have enough weight values equal to zero for a first set of PDPCCs to which the filter is assigned to compute dot products for nodes that use the filter, the method divides the filter between the first set and a second set of PDPCCs. The method generates program instructions for instructing the NNIC to execute the NN by using the first and second PDPCCs to compute dot products for the nodes that use the filter.
1. A method for compiling a neural network program for a neural network inference circuit that comprises a plurality of dot product cores that each comprises a same number of partial dot product computation circuits for computing dot products between weight values and input values, the method comprising:
receiving a neural network definition comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node;
assigning a group of filters to specific partial dot product computation circuits, each filter in the group assigned to a different set of the partial dot product computation circuits;
when a particular filter does not have enough weight values equal to zero for a first set of partial dot product computation circuits to which the particular filter is assigned to compute dot products for the computation nodes that use the particular filter, dividing the particular filter between the first set of partial dot product computation circuits and a second set of partial dot product computation circuits; and
generating program instructions for instructing the neural network inference circuit to execute the neural network by using the first and second partial dot product computation circuits to compute the dot products for the computation nodes that use the particular filter,
wherein each partial dot product computation circuit is assigned an index and each set of partial dot product computation circuits having a same index comprises one partial dot product computation circuit from each core that are connected by a separate dot product aggregation bus.
2. The method of claim 1 , wherein each separate dot product aggregation bus aggregates partial dot products from the partial dot product computation circuits that connect to the dot product aggregation bus.
3. The method of claim 1 , wherein assigning the particular filter to the first set of the partial dot product computation circuits comprises assigning the particular filter to partial dot product computation circuits having a same index in each core of a set of the cores.
4. The method of claim 3 , wherein dividing the particular filter comprises assigning a first subset of the weight values of the particular filter to the first set of partial dot product computation circuits and assigning a second subset of the weight values of the particular filter to the second set of partial dot product computation circuits.
5. The method of claim 4 , wherein:
the neural network inference circuit further comprises the same number of sets of post-processing circuits as partial dot product computation circuits in each core;
a particular post-processing circuit in the set of post-processing circuits having the same index as the first set of partial dot product computation circuits receives an aggregated dot product from the dot product aggregation bus connected to the first set of partial dot product computation circuits as a first input;
the particular post-processing circuit receives an aggregated dot product from the dot product aggregation bus connected to the second set of partial dot product computation circuits as a second input; and
the particular post-processing circuit adds the first and second inputs to compute the dot products for computation nodes that use the particular filter.
6. The method of claim 4 , wherein the particular filter is a first filter and a second filter is initially assigned to the second set of partial dot product computation circuits, the method further comprising re-assigning a second filter that was initially assigned to the second set of partial dot product computation circuits to a third set of partial dot product computation circuits.
7. The method of claim 4 , wherein the set of weight values of the particular filter is a first set of weight values, wherein dividing the particular filter further comprises:
assigning the value zero to a third set of weight values that replaces the second subset of weight values in the first set of partial dot product computation circuits; and
assigning the value zero to a fourth set of weight values that replaces the first subset of weight values in the second set of partial dot product computation circuits.
8. The method of claim 1 , wherein:
each partial dot product computation circuit comprises a set of input multiplexers that reduces a first number of input values for a set of computation nodes to a second number of input values for a particular computation node; and
each of a first number of input value buffer slots is wired to at least two of the input multiplexers of at least one partial dot product computation circuit.
9. The method of claim 8 , wherein:
assigning a particular filter to a set of the partial dot product computation circuits comprises assigning subsets of the weight values of the particular filter to each partial dot product computation circuit in the set; and
each weight value has a corresponding input value for each computation node that uses the particular filter that includes the weight value.
10. The method of claim 1 , wherein the neural network is trained so that the plurality of filters has at least a threshold percentage of weight values equal to zero.
11. A method for compiling a neural network program for a neural network inference circuit that comprises a plurality of partial dot product computation circuits for computing dot products between weight values and input values, the method comprising:
receiving a neural network definition comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node;
assigning a group of filters to specific partial dot product computation circuits, wherein (i) assigning a particular filter to a set of the partial dot product computation circuits comprises assigning subsets of the weight values of the particular filter to each partial dot product computation circuit in the set, (ii) each weight value has a corresponding input value for each computation node that uses the particular filter that includes the weight value, (iii) each partial dot product computation circuit comprises a set of input multiplexers that reduces a first number of input values for a set of computation nodes to a second number of input values for a particular computation node, and (iv) each of a first number of input value buffer slots is wired to at least two of the input multiplexers of at least one partial dot product computation circuit;
when the particular filter does not have enough weight values equal to zero for a first set of partial dot product computation circuits to which the particular filter is assigned to compute dot products for the computation nodes that use the particular filter, dividing the particular filter between the first set of partial dot product computation circuits and a second set of partial dot product computation circuits, wherein the particular filter does not have enough weight values equal to zero when at least one subset of the weight values has more non-zero values than each partial dot product computation circuit has input multiplexers; and
generating program instructions for instructing the neural network inference circuit to execute the neural network by using the first and second partial dot product computation circuits to compute the dot products for the computation nodes that use the particular filter.
12. A method for compiling a neural network program for a neural network inference circuit that comprises a plurality of partial dot product computation circuits for computing dot products between weight values and input values, the method comprising:
receiving a neural network definition comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node;
assigning a group of filters to specific partial dot product computation circuits, wherein (i) assigning a particular filter to a set of the partial dot product computation circuits comprises assigning subsets of the weight values of the particular filter to each partial dot product computation circuit in the set, (ii) each weight value has a corresponding input value for each computation node that uses the particular filter that includes the weight value, (iii) each partial dot product computation circuit comprises a set of input multiplexers that reduces a first number of input values for a set of computation nodes to a second number of input values for a particular computation node, and (iv) each of a first number of input value buffer slots is wired to at least two of the input multiplexers of at least one partial dot product computation circuit;
determining, for each subset of the weight values assigned to a particular partial dot product computation circuit, whether each of the input value buffer slots for the input values that correspond to the non-zero weight values of the subset can be mapped to different input multiplexers;
when the particular filter does not have enough weight values equal to zero for a first set of partial dot product computation circuits to which the particular filter is assigned to compute dot products for the computation nodes that use the particular filter, dividing the particular filter between the first set of partial dot product computation circuits and a second set of partial dot product computation circuits; and
generating program instructions for instructing the neural network inference circuit to execute the neural network by using the first and second partial dot product computation circuits to compute the dot products for the computation nodes that use the particular filter.
13. The method of claim 12 , wherein the particular filter does not have enough weight values equal to zero when, for at least one subset of the weight values of the particular filter, each of the input value buffer slots for the input values that correspond to the non-zero weight values of the subset cannot be mapped to different input multiplexers.
14. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit compiles a neural network program for a neural network inference circuit that comprises a plurality of dot product cores that each comprises a same number of partial dot product computation circuits for computing dot products between weight values and input values, the program comprising sets of instructions for:
receiving a neural network definition comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node;
assigning a group of filters to specific partial dot product computation circuits, each filter in the group assigned to a different set of the partial dot product computation circuits;
when a particular filter does not have enough weight values equal to zero for a first set of partial dot product computation circuits to which the particular filter is assigned to compute dot products for the computation nodes that use the particular filter, dividing the particular filter between the first set of partial dot product computation circuits and a second set of partial dot product computation circuits; and
generating program instructions for instructing the neural network inference circuit to execute the neural network by using the first and second partial dot product computation circuits to compute the dot products for the computation nodes that use the particular filter,
wherein each partial dot product computation circuit is assigned an index and each set of partial dot product computation circuits having a same index comprises one partial dot product computation circuit from each core that are connected by a separate dot product aggregation bus.
15. The non-transitory machine-readable medium of claim 14 , wherein:
the set of instructions for assigning the particular filter to the first set of the partial dot product computation circuits comprises a set of instructions for assigning the particular filter to partial dot product computation circuits having a same index in each core of a set of the cores; and
the set of instructions for dividing the particular filter comprises a set of instructions for assigning (i) a first subset of the weight values of the particular filter to the first set of partial dot product computation circuits and (ii) a second subset of the weight values of the particular filter to the second set of partial dot product computation circuits.
16. The non-transitory machine-readable medium of claim 15 , wherein:
the neural network inference circuit further comprises the same number of sets of post-processing circuits as partial dot product computation circuits in each core;
a particular post-processing circuit in the set of post-processing circuits having the same index as the first set of partial dot product computation circuits receives an aggregated dot product from the dot product aggregation bus connected to the first set of partial dot product computation circuits as a first input;
the particular post-processing circuit receives an aggregated dot product from the dot product aggregation bus connected to the second set of partial dot product computation circuits as a second input; and
the particular post-processing circuit adds the first and second inputs to compute the dot products for computation nodes that use the particular filter.
17. The non-transitory machine-readable medium of claim 15 , wherein the particular filter is a first filter and a second filter is initially assigned to the second set of partial dot product computation circuits, the program further comprising a set of instructions for re-assigning a second filter that was initially assigned to the second set of partial dot product computation circuits to a third set of partial dot product computation circuits.
18. The non-transitory machine-readable medium of claim 15 , wherein the set of weight values of the particular filter is a first set of weight values, wherein the set of instructions for dividing the particular filter further comprises sets of instructions for:
assigning the value zero to a third set of weight values that replaces the second subset of weight values in the first set of partial dot product computation circuits; and
assigning the value zero to a fourth set of weight values that replaces the first subset of weight values in the second set of partial dot product computation circuits.
19. The non-transitory machine-readable medium of claim 14 , wherein:
each partial dot product computation circuit comprises a set of input multiplexers that reduces a first number of input values for a set of computation nodes to a second number of input values for a particular computation node;
each of a first number of input value buffer slots is wired to at least two of the input multiplexers of at least one partial dot product computation circuit;
the set of instructions for assigning a particular filter to a set of the partial dot product computation circuits comprises a set of instructions for assigning subsets of the weight values of the particular filter to each partial dot product computation circuit in the set; and
each weight value has a corresponding input value for each computation node that uses the particular filter that includes the weight value.
20. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit compiles a neural network program for a neural network inference circuit that comprises a plurality of partial dot product computation circuits for computing dot products between weight values and input values, the program comprising sets of instructions for:
receiving a neural network definition comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node;
assigning a group of filters to specific partial dot product computation circuits, wherein (i) assigning a particular filter to a set of the partial dot product computation circuits comprises assigning subsets of the weight values of the particular filter to each partial dot product computation circuit in the set, (ii) each weight value has a corresponding input value for each computation node that uses the particular filter that includes the weight value, (iii) each partial dot product computation circuit comprises a set of input multiplexers that reduces a first number of input values for a set of computation nodes to a second number of input values for a particular computation node, and (iv) each of a first number of input value buffer slots is wired to at least two of the input multiplexers of at least one partial dot product computation circuit;
when the particular filter does not have enough weight values equal to zero for a first set of partial dot product computation circuits to which the particular filter is assigned to compute dot products for the computation nodes that use the particular filter, dividing the particular filter between the first set of partial dot product computation circuits and a second set of partial dot product computation circuits, wherein the particular filter does not have enough weight values equal to zero when at least one subset of the weight values has more non-zero values than each partial dot product computation circuit has input multiplexers; and
generating program instructions for instructing the neural network inference circuit to execute the neural network by using the first and second partial dot product computation circuits to compute the dot products for the computation nodes that use the particular filter.
21. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit compiles a neural network program for a neural network inference circuit that comprises a plurality of partial dot product computation circuits for computing dot products between weight values and input values, the program comprising sets of instructions for:
receiving a neural network definition comprising a plurality of computation nodes, each computation node of a set of the computation nodes comprising a dot product between (i) a set of weight values of a filter and (ii) a set of input values for the computation node;
assigning a group of filters to specific partial dot product computation circuits, wherein (i) assigning a particular filter to a set of the partial dot product computation circuits comprises assigning subsets of the weight values of the particular filter to each partial dot product computation circuit in the set, (ii) each weight value has a corresponding input value for each computation node that uses the particular filter that includes the weight value, (iii) each partial dot product computation circuit comprises a set of input multiplexers that reduces a first number of input values for a set of computation nodes to a second number of input values for a particular computation node, and (iv) each of a first number of input value buffer slots is wired to at least two of the input multiplexers of at least one partial dot product computation circuit;
determining, for each subset of the weight values assigned to a particular partial dot product computation circuit, whether each of the input value buffer slots for the input values that correspond to the non-zero weight values of the subset can be mapped to different input multiplexers;
when the particular filter does not have enough weight values equal to zero for a first set of partial dot product computation circuits to which the particular filter is assigned to compute dot products for the computation nodes that use the particular filter, dividing the particular filter between the first set of partial dot product computation circuits and a second set of partial dot product computation circuits; and
generating program instructions for instructing the neural network inference circuit to execute the neural network by using the first and second partial dot product computation circuits to compute the dot products for the computation nodes that use the particular filter.
22. The non-transitory machine-readable medium of claim 21 , wherein the particular filter does not have enough weight values equal to zero when, for at least one subset of the weight values of the particular filter, each of the input value buffer slots for the input values that correspond to the non-zero weight values of the subset cannot be mapped to different input multiplexers.