Batch processing in a neural network processor
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a respective neural network output for each of a plurality of inputs, the method comprising, for each of the neural network layers: receiving a plurality of inputs to be processed at the neural network layer; forming one or more batches of inputs from the plurality of inputs, each batch having a number of inputs up to the respective batch size for the neural network layer; selecting a number of the one or more batches of inputs to process, where a count of the inputs in the number of the one or more batches is greater than or equal to the respective associated batch size of a subsequent layer in the sequence; and processing the number of the one or more batches of inputs to generate the respective neural network layer output.
1. A method for performing neural network computations using a hardware circuit, the method comprising:
obtaining a first batch of inputs to be processed at a layer of a neural network, wherein the layer has an associated batch size;
obtaining weights for the layer, wherein the weights have an associated reuse value that defines an amount of reuse of the weights for the layer;
selecting, based on the batch size and the reuse value of the weights, a particular quantity of inputs in the first batch to be processed at the layer using the weights, wherein the particular quantity of inputs is selected so as to not exceed: i) the batch size for the layer, or ii) an amount that the weights are permitted to be reused based on the reuse value; and
processing, using the weights, the particular quantity of inputs to generate a layer output.
2. The method of claim 1 , further comprising:
selecting the particular quantity of inputs based on a threshold fetch time of accessing memory of the hardware circuit to obtain new weights for the layer.
3. The method of claim 1 , wherein selecting the particular quantity of inputs in the first batch comprises:
determining a number of times the hardware circuit is permitted to reuse weights for the layer before a compute time of reusing the weights the number of times with distinct activation inputs is at least equal to a fetch time of accessing new weight inputs for the layer.
4. The method of claim 3 , wherein selecting the particular quantity of inputs comprises:
selecting the particular quantity of inputs such that the compute time of reusing the weights does not exceed the fetch time of accessing new weight inputs for the layer.
5. The method of claim 1 , wherein the hardware circuit comprises an array of compute cells and processing the particular quantity of inputs to generate the layer output comprises:
reusing, by two or more respective compute cells in the array, the weights for the layer and an activation input in the particular quantity of inputs over a first processor clock cycle and a second subsequent processor clock cycle.
6. The method of claim 5 , wherein processing the particular quantity of inputs to generate the layer output comprises:
processing multiple independent activation inputs while reusing weights that are loaded in the array for a given layer of the neural network, the weights for the given layer being reused based on the reuse value.
7. The method of claim 1 , further comprising:
reusing the weights for the layer based on a processing speed of the hardware circuit, wherein reusing the weights for the layer comprises reusing a first portion of weights to process an input in the particular quantity of inputs while, in parallel, accessing a second portion of weights from memory based on the processing speed of the hardware circuit, the second portion of weights to be used in a subsequent computation.
8. The method of claim 7 , wherein the weight reuse value is based on at least one of:
a number of arithmetic units included in the hardware circuit; or
a number of channels included in a memory of the hardware circuit that is used to store multiple batches of inputs to be processed at one or more layers of the neural network.
9. A system for performing neural network computations using a hardware circuit, the system comprising:
a processor; and
a non-transitory machine-readable storage device for storing instructions that are executable by the processor to cause performance of operations comprising:
obtaining a first batch of inputs to be processed at a layer of a neural network, wherein the layer has an associated batch size;
obtaining weights for the layer, wherein the weights have an associated reuse value that defines an amount of reuse of the weights for the layer;
selecting, based on the batch size and the reuse value of the weights, a particular quantity of inputs in the first batch to be processed at the layer using the weights, wherein the particular quantity of inputs is selected so as to not exceed: i) the batch size for the layer, or ii) an amount that the weights are permitted to be reused based on the reuse value; and
processing, using the weights, the particular quantity of inputs to generate a layer output.
10. The system of claim 9 , wherein the operations further comprise:
selecting the particular quantity of inputs based on a threshold fetch time of accessing memory of the hardware circuit to obtain new weights for the layer.
11. The system of claim 9 , wherein selecting the particular quantity of inputs in the first batch comprises:
determining a number of times the hardware circuit is permitted to reuse weights for the layer before a compute time of reusing the weights the number of times with distinct activation inputs is at least equal to a fetch time of accessing new weight inputs for the layer.
12. The system of claim 11 , wherein selecting the particular quantity of inputs comprises:
selecting the particular quantity of inputs such that the compute time of reusing the weights does not exceed the fetch time of accessing new weight inputs for the layer.
13. The system of claim 9 , wherein the hardware circuit comprises an array of compute cells and processing the particular quantity of inputs to generate the layer output comprises:
reusing, by two or more respective compute cells in the array, the weights for the layer and an activation input in the particular quantity of inputs over a first processor clock cycle and a second subsequent processor clock cycle.
14. The system of claim 13 , wherein processing the particular quantity of inputs to generate the layer output comprises:
processing multiple independent activation inputs while reusing weights that are loaded in the array for a given layer of the neural network, the weights for the given layer being reused based on the reuse value.
15. The system of claim 9 , further comprising:
reusing the weights for the layer based on a processing speed of the hardware circuit, wherein reusing the weights for the layer comprises reusing a first portion of weights to process an input in the particular quantity of inputs while, in parallel, accessing a second portion of weights from memory based on the processing speed of the hardware circuit, the second portion of weights to be used in a subsequent computation.
16. The system of claim 15 , wherein the weight reuse value is based on at least one of:
a number of arithmetic units included in the hardware circuit; or
a number of channels included in a memory of the hardware circuit that is used to store multiple batches of inputs to be processed at one or more layers of the neural network.
17. A non-transitory machine-readable storage device for storing instructions to perform neural network computations using a hardware circuit, the instructions being executable by a processor to cause performance of operations comprising:
obtaining a first batch of inputs to be processed at a layer of a neural network, wherein the layer has an associated batch size;
obtaining weights for the layer, wherein the weights have an associated reuse value that defines an amount of reuse of the weights for the layer;
selecting, based on the batch size and the reuse value of the weights, a particular quantity of inputs in the first batch to be processed at the layer using the weights, wherein the particular quantity of inputs is selected so as to not exceed: i) the batch size for the layer, or ii) an amount that the weights are permitted to be reused based on the reuse value; and
processing, using the weights, the particular quantity of inputs to generate a layer output.
18. The machine-readable storage device of claim 17 , wherein the operations further comprise:
selecting the particular quantity of inputs based on a threshold fetch time of accessing memory of the hardware circuit to obtain new weights for the layer.
19. The machine-readable storage device of claim 17 , wherein selecting the particular quantity of inputs in the first batch comprises:
determining a number of times the hardware circuit is permitted to reuse weights for the layer before a compute time of reusing the weights the number of times with distinct activation inputs is at least equal to a fetch time of accessing new weight inputs for the layer.
20. The machine-readable storage device of claim 17 , wherein the hardware circuit comprises an array of compute cells and processing the particular quantity of inputs to generate the layer output comprises:
reusing, by two or more respective compute cells in the array, the weights for the layer and an activation input in the particular quantity of inputs over a first processor clock cycle and a second subsequent processor clock cycle.