Neural network inference circuit read controller with multiple operational modes
Some embodiments provide a neural network inference circuit for executing a neural network with multiple layers. The neural network inference circuit includes a set of processing circuits for executing the layers of the neural network, a set of memories for storing data used by the set of processing circuits to execute the neural network layers, and a read controller for retrieving the data from the set of memories and storing the data in a cache for use by the set of processing circuits. The read controller retrieves the data in one of (i) a first mode for retrieving the data from sequential memory locations within the set of memories to store in the cache and (ii) a second mode for retrieving the data from non-sequential memory locations within the set of memories to store in the cache.
1. A neural network inference circuit for executing a neural network that comprises a plurality of layers, the neural network inference circuit comprising:
a set of processing circuits for executing the layers of the neural network;
a set of memories for storing data used by the set of processing circuits to execute the neural network layers; and
a read controller for retrieving the data from the set of memories and storing the data in a cache for use by the set of processing circuits,
wherein the read controller retrieves the data in one of (i) a first mode for retrieving the data from sequential memory locations within the set of memories to store in the cache and (ii) a second mode for retrieving the data from non-sequential memory locations within the set of memories to store in the cache,
wherein when the read controller retrieves the data in the first mode, the read controller performs a plurality of read operations over a plurality of clock cycles to build up the cache prior to providing the data from the cache to the set of processing circuits.
2. The neural network inference circuit of claim 1 , wherein each of a set of the layers comprises a plurality of computation nodes comprising a dot product of input values and weight values computed by the set of processing circuits.
3. The neural network inference circuit of claim 2 , wherein the read controller (i) retrieves the weight values for a particular layer using the first mode for sequential retrieval and (ii) retrieves the input values for the particular layer using the second mode for non-sequential retrieval.
4. The neural network inference circuit of claim 3 , wherein the weight values for the particular layer are loaded from the cache into a set of buffers to be used for a plurality of different dot products for a plurality of different computation nodes of the layer.
5. The neural network inference circuit of claim 4 , wherein:
the weight values are stored in memory as blocks of encoded weight data;
each block comprises the encoded weight data for one buffer in the set of buffers; and
each block of encoded weight data is read once from memory during each execution of the neural network.
6. The neural network inference circuit of claim 1 , wherein after the cache is built up, a latency for providing data to the set of processing circuits is less when the read controller retrieves the data in the first mode than when the read controller retrieves the data in the second mode.
7. A neural network inference circuit for executing a neural network that comprises a plurality of layers, the neural network inference circuit comprising:
a set of processing circuits for executing the layers of the neural network;
a set of memories for storing data used by the set of processing circuits to execute the neural network layers,
wherein each of a set of the layers comprises a plurality of computation nodes comprising a dot product of input values and weight values computed by the set of processing circuits;
a read controller for retrieving the data from the set of memories and storing the data in a cache for use by the set of processing circuits,
wherein the read controller retrieves the data in one of (i) a first mode for retrieving the data from sequential memory locations within the set of memories to store in the cache and (ii) a second mode for retrieving the data from non-sequential memory locations within the set of memories to store in the cache,
wherein the read controller (i) retrieves the weight values for a particular layer using the first mode for sequential retrieval and (ii) retrieves the input values for the particular layer using the second mode for non-sequential retrieval; and
a first set of buffers,
wherein the weight values for the particular layer are loaded into the first set of buffers and remain in the first set of buffers while the input values for the particular layer are iteratively retrieved, loaded into a second set of buffers, and used for a set of dot products for a set of the computation nodes of the particular layer.
8. The neural network inference circuit of claim 7 , wherein a particular set of the input values stored at a particular memory location in the set of memories are retrieved from the set of memories more than once during the execution of the particular layer.
9. The neural network inference circuit of claim 7 , wherein each time the particular set of input values are retrieved from the set of memories, the particular set of input values are stored in the cache and different subsets of the particular set of input values are loaded into the second set of buffers at different times.
10. A neural network inference circuit for executing a neural network that comprises a plurality of layers, the neural network inference circuit comprising:
a set of processing circuits for executing the layers of the neural network;
a set of memories for storing data used by the set of processing circuits to execute the neural network layers,
wherein each of a first set of the layers comprises a plurality of computation nodes comprising a dot product of input values and weight values computed by the set of processing circuits,
wherein each layer of a second set of layers comprises a plurality of computation nodes comprising element-wise operations performed on sets of input values by the set of processing circuits;
wherein a first subset of the set of processing circuits computes the dot products and a second, different subset of the set of processing circuits performs the element-wise operations; and
a read controller for retrieving the data from the set of memories and storing the data in a cache for use by the set of processing circuits,
wherein the read controller retrieves the data in one of (i) a first mode for retrieving the data from sequential memory locations within the set of memories to store in the cache and (ii) a second mode for retrieving the data from non-sequential memory locations within the set of memories to store in the cache.
11. The neural network inference circuit of claim 10 , wherein the read controller (i) retrieves the input values for the first set of layers using the second mode for non-sequential retrieval and (ii) retrieves the input values for the second set of layers using the first mode for sequential retrieval.
12. The neural network inference circuit of claim 11 , wherein:
each layer of a third set of layers comprises a plurality of computation nodes comprising pooling operations performed on sets of input values by the second subset of the set of processing circuits; and
the read controller retrieves the input values for the third set of layers using the second mode for non-sequential retrieval.
13. The neural network inference circuit of claim 1 , wherein the set of memories comprises banks of random access memory (RAM), wherein the memory locations comprise RAM words.
14. The neural network inference circuit of claim 1 , wherein each memory location has a fixed width storing a fixed amount of data.
15. The neural network inference circuit of claim 14 , wherein the cache comprises a plurality of rows, each row having the same fixed width as one memory location.
16. The neural network inference circuit of claim 15 , wherein the cache comprises a plurality of registers, wherein a latency to load data from the cache is less than a latency to retrieve data from the set of memories.
17. The neural network inference circuit of claim 15 , wherein for a particular clock cycle, the read controller provides up to the fixed width of data to the set of processing circuits.
18. The neural network inference circuit of claim 17 , wherein the read controller provides different amounts of data to the set of processing circuits depending on a pattern with which the data is stored in the set of memories.
19. The neural network inference circuit of claim 17 , wherein an amount of data provided by the read controller in the particular clock cycle is based on configuration data received by the read controller for the particular clock cycle.