Executing replicated neural network layers on inference circuit
Some embodiments provide a method for executing a layer of a neural network, for a circuit that restricts a number of weight values used per layer. The method applies a first set of weights to a set of inputs to generate a first set of results. The first set of weights are restricted to a first set of allowed values. For each of one or more additional sets of weights, the method applies the respective additional set of weights to the same set of inputs to generate a respective additional set of results. The respective additional set of weights is restricted to a respective additional set of allowed values that is related to the first set of allowed values and the other additional sets of allowed values. The method generates outputs for the particular layer by combining the first set of results with each respective additional set of results.
1 . A method for executing a particular layer of a neural network comprising a plurality of layers, the method comprising:
at a neural network inference circuit comprising first computation circuitry that restricts a number of weight values used for a set of computations to a first number of weight values:
applying, at a first layer, a first plurality of weight values to a plurality of input values to generate a first plurality of result values, wherein the first plurality of weight values are restricted to a first set of allowed values according to a first weight constraint and the first number of weight values;
applying, at a second layer, a second plurality of weight values to the plurality of input values to generate a second plurality of result values, the second plurality of weight values being restricted to a second set of allowed values according to a second weight constraint and the first number of weight values, wherein the second plurality of weight values are different than the first set of allowed values; and
generating, by the first computation circuitry, output values for the particular layer by combining the first plurality of result values with the second plurality of result values, wherein combining the first plurality of result values with the second plurality of result values enables the first computation circuitry to generate the output values for the particular layer based on a particular number of weight values that exceeds the first number of weight values restricted by the first computation circuitry.
2 . The method of claim 1 , further comprising:
computing the plurality of input values as output values of a particular layer; and
storing the plurality of input values in a first set of memories of the neural network inference circuit.
3 . The method of claim 2 , wherein:
the neural network inference circuit comprises a plurality of cores, each core comprising a plurality of memories; and
the first set of memories belong to a first set of the plurality of cores.
4 . The method of claim 3 , wherein data for the first plurality of weight values is stored in a second set of memories belonging to the first set of the plurality of cores.
5 . The method of claim 3 , further comprising:
storing the first plurality of result values in a second set of memories of the neural network inference circuit that belong to a second set of cores; and
storing each respective additional plurality of result values in a respective set of memories that belong to the second set of cores.
6 . The method of claim 5 , wherein:
each result value from the first plurality of result values is combined with one result value from each respective additional plurality of result values; and
each set of result values that is combined is stored in a same core.
7 . The method of claim 5 , further comprising storing the output values for the particular layer in a third set of memories of the neural network inference circuit that belong to a third set of cores.
8 . The method of claim 1 , wherein the neural network inference circuit (i) receives instructions for applying the first plurality of weight values to the plurality of input values as instructions for a first layer and (ii) receives instructions for applying each additional plurality of weight values to the plurality of input values as instructions for a separate additional layer.
9 . The method of claim 8 , wherein the neural network inference circuit receives instructions for generating the output values for the particular layer by combining the first plurality of result values with each respective additional plurality of result values as instructions for one or more additional layers.
10 . An inference circuit for executing a particular layer of a neural network comprising a plurality of layers, the inference circuit comprising:
first computation circuitry configured to:
restrict a number of weight values used to a first number of weight values for a set of computations;
apply, at a first layer, a first plurality of weight values to a plurality of input values to generate a first plurality of result values, wherein the first plurality of weight values are restricted to a first set of allowed values according to a first weight constraint and the number of weight values;
apply, at a second layer, a second plurality of weight values to the plurality of input values to generate a second plurality of result values, the second plurality of weight values being restricted to a second set of allowed values according to a second weight constraint and the number of weight values, wherein the second plurality of weight values are different than the first set of allowed values; and
generate output values for the particular layer by combining the first plurality of result values with the second plurality of result values, wherein combining the first plurality of result values with the second plurality of result values enables the first computation circuitry to generate the output values for the particular layer based on a particular number of weight values that exceeds the first number of weight values.
11 . The inference circuit of claim 10 , wherein the first computation circuitry is further configured to:
compute the plurality of input values as output values of a particular layer; and
store the plurality of input values in a first set of memories.
12 . The inference circuit of claim 11 , wherein data for the first plurality of weight values is stored in a second set of memories belonging to a first set of cores.
13 . The inference circuit of claim 10 , wherein the first computation circuitry is further configured to:
store the first plurality of result values in a second set of memories that belong to a second set of cores; and
store each respective additional plurality of result values in a respective set of memories that belong to the second set of cores.
14 . The inference circuit of claim 13 , wherein:
each result value from the first plurality of result values is combined with one result value from each respective additional plurality of result values; and
each set of result values that is combined is stored in a same core.
15 . The inference circuit of claim 13 , wherein the first computation circuitry is further configured to:
store the output values for the particular layer in a third set of memories that belong to a third set of cores.