Neural network inference circuit performing matrix multiplication
Some embodiments provide a neural network inference circuit (NNIC) for executing a network having multiple layers. The NNIC includes multiple circuit sets. Each circuit set includes a dot product circuit to compute dot products between weight values and activation values for at least a subset of a first set of the layers, a math function circuit to compute values based on computations using activation values for at least a subset of a second set of the layers, and a post-processing circuit to receive (i) values output by the dot product circuit and (ii) values output by the math function circuit and to perform post-processing operations on the received values. The NNIC includes a set of accumulation circuits. Each accumulation circuit is to accumulate outputs of math function circuits for layers of the second set of layers that perform matrix multiplication of sets of activation values output by previous layers.
1 . A neural network inference circuit for executing a neural network comprising a plurality of layers, the neural network inference circuit comprising:
a plurality of clusters, each cluster comprising a respective plurality of circuit sets, each circuit set of the respective plurality of circuit sets comprising:
a dot product circuit to compute and output dot products between weight values and activation values for at least a subset of a first set of the plurality of layers;
a math function circuit to compute and output values based on computations using activation values for at least a subset of a second set of the plurality of layers; and
a post-processing circuit to receive (i) values output by the dot product circuit for the subset of the first set of the plurality of layers and (ii) values output by the math function circuit for the subset of the second set of the plurality of layers and to perform post-processing operations on the received values; and
a set of accumulation circuits, each accumulation circuit to accumulate outputs of a plurality of math function circuits included in the respective plurality of circuit sets for layers of the second set of the plurality of layers that perform matrix multiplication of a first set of activation values output by a previous neural network layer with a second set of activation values output by another previous neural network layer.
2 . The neural network inference circuit of claim 1 further comprising a plurality of cores to store activation values.
3 . The neural network inference circuit of claim 2 , wherein each cluster of the plurality of clusters comprises a respective set of the plurality of cores.
4 . The neural network inference circuit of claim 3 , wherein the respective set of the plurality of cores (i) perform dot product computations between weight values stored in the respective set of the plurality of cores and the activation values stored in the respective set of the plurality of cores and (ii) provide results of the dot product computations to a dot product bus that accumulates the results from the respective set of the plurality of cores in a first and the respective set of the plurality of cores in a second cluster.
5 . The neural network inference circuit of claim 4 , wherein the dot product bus comprises a plurality of lanes, each respective lane connecting to a respective dot product circuit of a respective circuit set in each of the plurality of clusters and providing accumulated dot product computation results to respective circuit sets.
6 . The neural network inference circuit of claim 4 , wherein the math function circuit receives activation values directly from the respective set of the plurality of cores that the math function circuit is associated with.
7 . The neural network inference circuit of claim 1 , wherein a particular layer of the second set of the plurality of layers comprises a matrix multiplication of activation values of a first layer by activation values of a second layer.
8 . The neural network inference circuit of claim 7 , wherein during execution of the particular layer, each math function circuit of a set of math function circuits receives a respective first activation value from the first layer and a respective second activation value from the second layer.
9 . The neural network inference circuit of claim 8 , wherein:
the respective first activation value is received by each math function circuit in the set of math function circuits during a first clock cycle of the neural network inference circuit;
the respective second activation value is received by each math function circuit in the set of math function circuits during a second clock cycle of the neural network inference circuit; and
each math function circuit of the set of math function circuits multiplies the respective first activation value by the respective second activation value during the second clock cycle.
10 . The neural network inference circuit of claim 8 , wherein a particular one of the set of accumulation circuits adds outputs of the set of math function circuits.
11 . The neural network inference circuit of claim 10 , wherein:
each output value of the particular layer is based on a particular number of multiplications between activation values from the first layer and activation values from the second layer; and
when the particular number of multiplications is larger than a number of math function circuits that provide their outputs to the particular one of the set of accumulation circuits, each math function circuit in the set of math function circuits performs a plurality of multiplications between activation values from the first layer and activation values from the second layer that are accumulated together by the particular one of the set of accumulation circuits to generate an output value of the particular layer.
12 . The neural network inference circuit of claim 11 , wherein the particular one of the set of accumulation circuits comprises a register for storing intermediate accumulated values.
13 . The neural network inference circuit of claim 10 , wherein each accumulation circuit in the set of accumulation circuits receives outputs from a same number of math function circuits during execution of the particular layer.
14 . The neural network inference circuit of claim 8 , wherein:
execution of the particular layer generates a plurality of channels of output activation values;
first output activation channel is based on multiplication of a first channel of the first layer by a first channel of the second layer; and
a second output activation channel is based on multiplication of a second channel of the first layer by a second channel of the second layer.
15 . The neural network inference circuit of claim 14 , wherein:
a first subset of the set of math function circuits receives activation values from the first channel of the first layer and the first channel of the second layer;
a second subset of the set of math function circuits receives activation values from the second channel of the first layer and the second channel of the second layer;
a first accumulation circuit accumulates outputs of each math function circuit in the first subset of the set of math function circuits; and
a second accumulation circuit accumulates outputs of each math function circuit in the second subset of the set of math function circuits.
16 . The neural network inference circuit of claim 14 , wherein the activation values of the first channel of the first layer and the first channel of the second layer are stored in a first core of the neural network inference circuit and the activation values of the second channel of the first layer and the second channel of the second layer are stored in a second core of the neural network inference circuit.
17 . The neural network inference circuit of claim 1 , wherein the neural network comprises an attention mechanism.
18 . The neural network inference circuit of claim 17 , wherein at least one of the layers that perform matrix multiplication are part of the attention mechanism of the neural network.