Computation of neural network node by neural network inference circuit
Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network that includes multiple computation nodes at multiple layers. The NNIC includes a set of clusters of core computation circuits and a channel, connecting the core computation circuits, that includes separate segments corresponding to each of the clusters. The NNIC includes a fabric controller circuit, a cluster controller circuit for each of the clusters, and a core controller circuit for each of the core computation circuits. The fabric controller circuit receives high-level neural network instructions from a microprocessor and parses the high-level neural network instructions.
1. For a neural network inference circuit, a method for executing a neural network that comprises a plurality of computation nodes, each of a set of the computation nodes comprising a dot product of input values and weight values, the method comprising:
to compute a particular computation node:
at each respective dot product core circuit of a plurality of dot product core circuits of the neural network inference circuit, computing a respective partial dot product using a respective set of input values and a respective set of weight values stored in a respective set of memories of the respective dot product core circuit; and
at a bus of the neural network inference circuit that comprises a plurality of aggregation circuits, combining the partial dot products computed by the plurality of dot product core circuits to compute the dot product for the particular computation node.
2. The method of claim 1 further comprising:
to compute each computation node of a plurality of additional computation nodes:
at each respective dot product core circuit of the plurality of dot product core circuits, computing a respective partial dot product for the computation node using a same respective set of input values and a different respective set of weight values stored in the respective set of memories of the respective dot product core circuit; and
at the bus, combining the partial dot products computed by the at least two dot product core circuits to compute the dot product for the particular computation node.
3. The method of claim 2 , wherein for a particular dot product core circuit, a same set of input values is used for the particular computation node and each of the additional computation nodes.
4. The method of claim 3 , wherein for the particular dot product core circuit, different sets of weight values are used for the particular computation node and each of the additional computation nodes.
5. The method of claim 2 , wherein the neural network inference circuits computes the particular computation node and each of the additional computation nodes simultaneously.
6. The method of claim 5 , wherein the particular computation node and each of the additional computation nodes belongs to a same layer of the neural network.
7. The method of claim 2 , wherein the partial dot products for each of the computation nodes are combined by a different aggregation circuit of the bus.
8. The method of claim 2 further comprising, prior to computing the particular computation node and the plurality of additional computation nodes:
loading the respective sets of input values into respective input buffers of the respective dot product core circuits from the respective sets of memories; and
loading the respective sets of weight values into respective pluralities of weight buffers of the respective dot product core circuits from the respective sets of memories.
9. The method of claim 8 , wherein the input values are loaded into a respective single input buffer in each respective dot product core circuit and the weight values are loaded into a respective plurality of weight buffers in each respective dot product core circuit.
10. The method of claim 9 , wherein a number of weight buffers into which the weight values are loaded in each dot product core circuit is the same as a number of computation nodes comprised by the particular computation node and the plurality of additional computation nodes.
11. The method of claim 9 , wherein each respective dot product core circuit comprises a respective plurality of adder tree circuits, wherein each respective adder tree circuit computes a respective partial dot product for one of the computation nodes by using the input values from the respective input buffer of the respective dot product core circuit and the weight values from one of the weight buffers of the respective dot product core circuit.
12. The method of claim 1 , wherein each of the dot product core circuits comprises a same number of adder tree circuits, each adder tree circuit having an index.
13. The method of claim 12 , wherein the partial dot products for the particular computation node are computed in the plurality of dot product core circuits by adder tree circuits having a same index.
14. The method of claim 13 , wherein the adder tree circuits having the same index in different dot product core circuits provide their computed partial dot products to a same aggregation circuit of the bus.
15. The method of claim 12 , wherein each respective aggregation circuit of the bus corresponds to respective adder tree circuits of each of the dot product core circuits having a same index.
16. The method of claim 1 , wherein each respective set of memories comprises one or more banks of random access memory (RAM), wherein a first block of the RAM for a particular dot product core circuit is allocated to the set of input values used by the particular dot product core circuit and a second block of the RAM for the particular dot product core circuit is allocated to the set of weight values used by the particular dot product core circuit.
17. The method of claim 16 , wherein the RAM for the particular dot product core circuit stores (i) all of the weight values used by the particular dot product core circuit for the entire neural network and (ii) input values for up to two layers of the neural network.
18. The method of claim 17 , wherein the input values for the particular computation node are overwritten by input values for subsequent nodes of the neural network.
19. The method of claim 1 further comprising, to compute the particular computation node, performing a set of post-processing operations on the dot product for the particular computation node at a set of post-processing circuits of the neural network inference circuit.
20. The method of claim 19 , wherein performing the set of post-processing operations comprises:
performing a bias operation and a scaling operation to the dot product at a first subset of the post-processing circuits; and
applying a non-linear activation function to an output of the first subset of the post-processing circuits at a second subset of the post-processing circuits.