Computation of neural network node with large input values
Some embodiments provide a method for a circuit that executes a neural network including multiple nodes. The method loads a set of weight values for a node into a set of weight value buffers, a first set of bits of each input value of a set of input values for the node into a first set of input value buffers, and a second set of bits of each of the input values into a second set of input value buffers. The method computes a first dot product of the weight values and the first set of bits of each input value and a second dot product of the weight values and the second set of bits of each input value. The method shifts the second dot product by a particular number of bits and adds the first dot product with the bit-shifted second dot product to compute a dot product for the node.
1. For a neural network inference circuit that executes a neural network comprising a plurality of computation nodes, the neural network inference circuit comprising a plurality of dot product cores for computing partial dot products, each of a set of the plurality of computation nodes comprising a dot product of input values and ternary weight values, a method for computing an output value for a particular computation node, the method comprising:
at each core of a set of the plurality of dot product cores of the neural network inference circuit, loading (i) data for a set of ternary weight values for the particular computation node into a weight value buffer of the core, (ii) a first portion of each input value of a set of input values for the particular computation node into a first input value buffer of the core, and (iii) a second portion of each of the input values into a second input value buffer of the core;
at a set of dot product computation circuits of the neural network inference circuit, the set of dot product computation circuits comprising a partial dot product computation circuit of each core of the set of dot product cores:
computing a first dot product between the set of ternary weight values from the weight value buffers of the set of cores and the first portion of each of the input values from the first input value buffers of the set of cores;
computing a second dot product between the set of ternary weight values and the second portion of each of the input values from the second input value buffers of the set of cores;
bit-shifting the second dot product to generate a bit-shifted second dot product; and
adding the first dot product with the bit-shifted second dot product to generate a computed dot product for the particular computation node; and
at a set of post-processing circuits of the neural network inference circuit, performing a set of post-processing operations to compute the output value for the particular computation node from the computed dot product for the particular computation node.
2. The method of claim 1 , wherein the first portion of each of the input values comprises a particular number of bits by which the second dot product is bit-shifted.
3. The method of claim 1 , wherein the first portion of each input value is a set of least significant bits of the input value and the second portion of each input value is a set of most significant bits of the input value.
4. The method of claim 1 , wherein (i) each input value is an 8-bit value, (ii) the first portion of each input value comprises a least significant 4 bits of the input value, and (iii) the second portion of each input value comprises a most significant 4 bits of the input value.
5. The method of claim 4 , wherein each of the first input value buffer and each second input value buffer comprises a set of slots for storing input values, wherein each slot stores 4 bits.
6. The method of claim 1 , wherein the partial dot product computation circuits of the set of dot product cores perform partial dot product computations of the first dot product and the second dot product.
7. The method of claim 6 , wherein a dot product bus of the neural network inference circuit aggregates the partial dot product computations of the first dot product and aggregates the partial dot product computations of the second dot product.
8. The method of claim 7 , wherein the dot product bus provides the first dot product and the second dot product to a dot product processing circuit of the neural network inference circuit that shifts the second dot product and adds the first dot product with the bit-shifted second dot product.
9. The method of claim 8 , wherein the dot product processing circuit provides the bit-shifted second dot product to the set of post-processing circuits.
10. The method of claim 9 , wherein:
each dot product core of the neural network inference circuit comprises a plurality of sets of weight value buffers and a plurality of partial dot product computation circuits for simultaneously computing partial dot products for different computation nodes;
the dot product bus comprises a plurality of independent aggregation circuits for aggregating partial dot products for different simultaneously-computed computation nodes; and
the neural network inference circuit comprises (i) a plurality of dot product processing circuits for simultaneously adding first dot products with bit-shifted second dot products for the different simultaneously-computed computation nodes and (ii) a plurality of sets of post-processing circuits for simultaneously performing sets of post-processing operations to compute output values for the different simultaneously-computed computation nodes.
11. The method of claim 1 , wherein the first dot product and the second dot product are computed in different clock cycles of the neural network inference circuit.
12. The method of claim 11 , wherein the set of dot product computation circuits computes the first dot product in a first clock cycle, the method further comprising storing the first dot product in a register of the set of dot product computation circuits for at least one clock cycle.
13. The method of claim 12 , wherein the set of dot product computation circuits computes the second dot product in a second clock cycle that is after the first clock cycle.
14. The method of claim 13 , wherein the set of dot product computation circuits bit-shifts the second dot product and adds the first dot product from the register with the bit-shifted second dot product in the second clock cycle.
15. The method of claim 11 , wherein the set of dot product computation circuits computes the second dot product and bit-shifts the second dot product in a first clock cycle, the method further comprising storing the bit-shifted second dot product in a register for at least one clock cycle.
16. The method of claim 15 , wherein the set of dot product computation circuits computes the first dot product in a second clock cycle that is after the first clock cycle.
17. The method of claim 16 , wherein the set of dot product computation circuits adds the bit-shifted second dot product from the register with the first dot product in the second clock cycle.
18. The method of claim 1 , wherein each ternary weight value is one of a positive value, a negation of the positive value, and zero.
19. The method of claim 1 , wherein performing the set of post-processing operations comprises:
at an adder circuit, adding a bias value to the computed dot product to compute a first intermediate result;
at a multiplier circuit, multiplying the first intermediate result by a scaling value to compute a second intermediate result; and
at a non-linear activation function circuit, applying a non-linear activation function to the second intermediate result to compute the output value for the particular computation node.