Non-dot product computations on neural network inference circuit
Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network. The NNIC includes a first circuit that outputs dot products for computation nodes of a first set of neural network layers, that include dot product computations of sets of weight values with sets of input values. The NNIC also includes a second circuit that outputs values for computation nodes of a second set of neural network layers, that apply a set of calculations that do not include dot products to sets of input values. The NNIC also includes a selection circuit that selects a dot product output from the first circuit when a current layer being processed by the NNIC belongs to the first set of layers, and selects a non-dot product output from the second circuit when the current layer belongs to the second set of layers.
1. A neural network inference circuit for executing a neural network comprising a plurality of layers of computation nodes, the neural network inference circuit comprising:
a first circuit that outputs dot products for computation nodes of a first set of neural network layers, said first set of layers comprising computation nodes that include dot product computations of sets of weight values with sets of input values;
a second circuit that outputs values for computation nodes of a second set of neural network layers, said second set of layers comprising computation nodes that apply a set of calculations that do not include dot products to sets of input values; and
a selection circuit that selects (i) a dot product output from the first circuit when a current layer being processed by the neural network inference circuit belongs to the first set of layers and (ii) a non-dot product output from the second circuit when the current layer belongs to the second set of layers.
2. The neural network inference circuit of claim 1 , wherein the second set of layers comprises a pooling layer, wherein each computation node of the pooling layer combines a different plurality of the input values of the layer such that the pooling layer has fewer output values than input values.
3. The neural network inference circuit of claim 2 , wherein each computation node of the pooling layer averages its respective plurality of input values.
4. The neural network inference circuit of claim 2 , wherein each computation node of the pooling layer identifies a maximum value of its respective plurality of input values.
5. The neural network inference circuit of claim 1 , wherein the second set of layers comprises an element-wise operation layer, wherein each computation node of the element-wise operation layer combines a first input value that is an output value of a first particular layer and a second input value that is an output value of a second particular layer.
6. The neural network inference circuit of claim 5 , wherein combining the first and second input values comprises adding the first and second input values.
7. The neural network inference circuit of claim 5 , wherein combining the first and second input values comprises multiplying the first and second input values.
8. The neural network inference circuit of claim 5 , wherein the first particular layer and second particular layer each belong to the first set of layers and each have a same number of computation nodes.
9. The neural network inference circuit of claim 1 , wherein the selection circuit is a multiplexer that receives data from each of the first and second circuits and selects the data from one of the circuits based on configuration data.
10. The neural network inference circuit of claim 9 , wherein the configuration data is received from a controller circuit.
11. The neural network inference circuit of claim 1 , wherein the first circuit, second circuit, and selection circuit are part of a post-processing circuit of the neural network inference circuit.
12. The neural network inference circuit of claim 11 further comprising a plurality of post-processing circuits that each comprises a corresponding first circuit, second circuit, and selection circuit.
13. The neural network inference circuit of claim 12 , wherein each respective selection circuit of a respective post-processing circuit in the plurality of post-processing circuits makes a same selection of the respective non-dot product output from the respective second circuit when the current layer belongs to the second set of layers.
14. The neural network inference circuit of claim 13 , wherein each of the selection circuits makes the same selection based on receiving a same set of configuration data.
15. The neural network inference circuit of claim 11 , wherein the post-processing circuit further comprises a third circuit for performing additional post-processing operations on the output selected by the selection circuit.
16. The neural network inference circuit of claim 15 , wherein the third circuit comprises (i) an adder circuit for adding a first predetermined value to the selected output and (ii) a multiplier circuit for multiplying an output of the adder circuit by a second predetermined value.
17. The neural network inference circuit of claim 15 , wherein the third circuit comprises a set of circuits that set an output of the post-processing circuit to a desired precision and number of bits.
18. The neural network inference circuit of claim 17 , wherein the output of the post-processing circuit is transported to a memory of the neural network inference circuit for storage.
19. The neural network inference circuit of claim 1 further comprising a plurality of cores for (i) computing partial dot products to provide to the first circuit and (ii) storing the sets of input values.
20. The neural network inference circuit of claim 19 , wherein the cores provide sets of input values for layers of the second set of layers directly to the second circuit.