Time-multiplexed dot products for neural network inference circuit
Some embodiments provide a method for a neural network inference circuit that executes a neural network including multiple nodes. The method loads a first set of weight values into a first set of weight value buffers, a second set of weight values into a second set of weight value buffers, a first set of input values into a first set of input value buffers, and a second set of input values into a second set of input value buffers. In a first clock cycle, the method computes a first dot product of the first set of weight values and the first set of input values. In a second clock cycle, the method computes a second dot product of the second set of weight values and the second set of input values. The method adds the first and second dot products to compute a dot product for the node.
1. For a neural network inference circuit that executes a neural network comprising a plurality of computation nodes, each of a set of the computation nodes comprising a dot product of input values and weight values, a method for computing a dot product for a computation node, the method comprising:
loading (i) a first set of weight values for the computation node into a first set of weight value buffers, (ii) a second set of weight values for the computation node into a second set of weight value buffers, (iii) a first set of input values for the computation node into a first set of input value buffers, and (iv) a second set of input values for the computation node into a second set of input value buffers;
in a first clock cycle of the neural network inference circuit, computing a first dot product of the first set of weight values and the first set of input values;
in a second clock cycle of the neural network inference circuit, computing a second dot product of the second set of weight values and the second set of input values; and
adding the first and second dot products to compute the dot product for the computation node.
2. The method of claim 1 , wherein a same set of circuits of the neural network inference circuit computes the first dot product and the second dot product.
3. The method of claim 2 , wherein the set of circuits that computes the first and second dot products comprises:
a set of one or more partial dot product computation circuits that each computes a dot product of (i) a respective subset of input values from a respective input value buffer and (ii) a respective subset of weight values from a respective weight value buffer; and
a dot product bus for aggregating dot products from the one or more partial dot product computation circuits.
4. The method of claim 3 , wherein each partial dot product computation circuit computes a dot product of a respective subset of the first set of input values and a respective subset of the first set of weight values in the first clock cycle and computes a dot product of a respective subset of the second set of input values and a respective subset of the second set of weight values in the second clock cycle.
5. The method of claim 3 , wherein the neural network inference circuit comprises a plurality of cores, wherein each core comprises a plurality of partial dot product computation circuits for simultaneously computing dot products of the set of input values from an input value buffer of the core and different sets of weight values from different weight value buffers of the core.
6. The method of claim 5 , wherein the set of partial dot product computation circuits that computes the first and second dot products comprises one of the partial dot product computation circuits from each of a set of the cores.
7. The method of claim 6 , wherein the set of cores comprises more than half of the cores.
8. The method of claim 3 , wherein the neural network inference circuit further comprises a post-processing unit that adds the first and second dot products.
9. The method of claim 1 further comprising storing the first dot product in a register until the second clock cycle.
10. The method of claim 9 , wherein first and second dot products are added in the second clock cycle.
11. The method of claim 1 further comprising performing a set of post-processing operations on the dot product for the computation node in order to compute an output value for the computation node.
12. The method of claim 1 , wherein the computation node is a first computation node with more than a particular number of input values, wherein for a second computation node with less than the particular number of input values the neural network inference circuit loads the weight values for the second computation node into only weight buffers of the first set of weight value buffers and loads the input values of for the second computation node into only input value buffers of the first set of input value buffers.
13. The method of claim 12 , wherein the neural network inference circuit comprises a plurality of cores that each comprises a primary input value buffer and a secondary input value buffer, wherein the first set of weight value buffers is a set of the primary input value buffers of the cores.
14. The method of claim 13 , wherein the particular number is a number of input values stored by one of the primary input value buffers multiplied by a number of cores of the neural network inference circuit.
15. A neural network inference circuit that executes a neural network comprising a plurality of computation nodes, each of a set of the computation nodes comprising a dot product of input values and weight values, the neural network inference circuit comprising:
a set of memory control circuits to load (i) a first set of weight values for a computation node into a first set of weight value buffers, (ii) a second set of weight values for the computation node into a second set of weight value buffers, (iii) a first set of input values for the computation node into a first set of input value buffers, and (iv) a second set of input values for the computation node into a second set of input value buffers;
a set of dot product computation circuits to:
compute, in a first clock cycle of the neural network inference circuit, a first dot product of the first set of weight values and the first set of input values; and
compute, in a second clock cycle of the neural network inference circuit, a second dot product of the second set of weight values and the second set of input values; and
an adder circuit to add the first and second dot products to compute a dot product for the computation node.
16. The neural network inference circuit of claim 15 , wherein the set of dot product computation circuits comprises:
a set of one or more partial dot product computation circuits to each compute a dot product of (i) a respective subset of input values from a respective input value buffer and (ii) a respective subset of weight values from a respective weight value buffer; and
a dot product bus for aggregating dot products from the one or more partial dot product computation circuits.
17. The neural network inference circuit of claim 16 , wherein each partial dot product computation circuit computes a dot product of a respective subset of the first set of input values and a respective subset of the first set of weight values in the first clock cycle and computes a dot product of a respective subset of the second set of input values and a respective subset of the second set of weight values in the second clock cycle.
18. The neural network inference circuit of claim 16 further comprising a plurality of cores, wherein each core comprises a plurality of partial dot product computation circuits for simultaneously computing dot products of the set of input values from an input value buffer of the core and different sets of weight values from different weight value buffers of the core, wherein the set of partial dot product computation circuits that computes the first and second dot products comprises one of the partial dot product computation circuits from each of a set of the cores.
19. The neural network inference circuit of claim 15 further comprising a register to store the first dot product until the second clock cycle.
20. A non-transitory machine-readable medium storing a program, for execution by at least one processing unit, that configures a neural network inference circuit that executes a neural network comprising a plurality of computation nodes, each of a set of the computation nodes comprising a dot product of input values and weight values, the program comprising sets of instructions for:
configuring a set of memory control circuits to load (i) a first set of weight values for a computation node into a first set of weight value buffers, (ii) a second set of weight values for the computation node into a second set of weight value buffers, (iii) a first set of input values for the computation node into a first set of input value buffers, and (iv) a second set of input values for the computation node into a second set of input value buffers;
configuring a set of dot product computation circuits to:
compute, in a first clock cycle of the neural network inference circuit, a first dot product of the first set of weight values and the first set of input values; and
compute, in a second clock cycle of the neural network inference circuit, a second dot product of the second set of weight values and the second set of input values; and
configuring an adder circuit to add the first and second dot products to compute a dot product for the computation node.
21. The non-transitory machine-readable medium of claim 20 , wherein the adder circuit is configured to add the first and second dot products in the second clock cycle.
22. The non-transitory machine-readable medium of claim 20 , wherein the program further comprises a set of instructions for configuring a post-processing circuit to perform a set of post-processing operations on the dot product for the computation node in order to compute an output value for the computation node.