Address decoding by neural network inference circuit read controller
Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network (NN) that includes computation nodes at multiple layers. The NNIC includes a set of processing circuits for executing the computation nodes of the NN, a set of memories for storing data used by the processing circuits to execute the NN layers, and a read controller for retrieving the data from the memories for use by the processing circuits. The data is stored in the memories as multiple varying-size blocks. The read controller receives read instructions for a requested block of data to be used by the processing circuits for one or more computation nodes. The read instructions include a base memory address for multiple blocks of data, a size of the requested block of data, and a location of the requested block of data within the multiple blocks of data.
1 . A neural network inference circuit for executing a neural network, the neural network comprising plurality of computation nodes at a plurality of layers, each of a set of the computation nodes comprising a dot product of input values and weight values, the neural network inference circuit comprising:
a set of processing circuits for executing the computation nodes of the neural network;
a set of memories for storing a plurality of varying-size blocks of input values and encoded weight data used by the set of processing circuits to execute the neural network layers;
a read controller for retrieving the data from the set of memories for use by the set of processing circuits; and
a first computing component for (i) providing instructions to the set of processing circuits and the read controller and (ii) providing data retrieved from the set of memories by the read controller to the set of processing circuits for the set of processing circuits to execute the computation nodes of the neural network,
wherein the read controller receives sets of read instructions for requested blocks of data to be used by the set of processing circuits for one or more computation nodes, each set of read instructions comprising (i) a base memory address for a plurality of blocks of data, (ii) a size of the requested block of data, and (iii) a location of the requested block of data within the plurality of blocks of data,
wherein the weight values for a particular layer are organized as a plurality of filters, each respective computation node of the particular layer computing a dot product of a respective group of input values to the particular layer and the weight values of a respective one of the filters, each filter stored in the set of memories as a separate block of encoded weight data, at least two different filters of the particular layer stored in the set of memories as two differently-sized blocks using different amounts of memory,
wherein a first set of read instructions for retrieving encoded weight data for a particular filter of the particular layer specifies a first block size, and wherein data retrieved using the first set of read instructions is used by the read controller to determine a second block size, wherein the second block size represents an amount by which the particular filter extends beyond a word boundary in the set of memories, and wherein a second set of read instructions for retrieving the encoded weight data for the particular filter specifies the second block size.
2 . The neural network inference circuit of claim 1 , wherein (i) the encoded weight data is stored in the set of memories at boot-up of the neural network inference circuit and (ii) the input values for each layer are stored in the set of memories during the execution of a previous layer.
3 . The neural network inference circuit of claim 2 , wherein for the particular layer, the size of the blocks of data is constant for blocks of the input values for the particular layer.
4 . The neural network inference circuit of claim 1 , wherein the plurality of filters are for convolutional operations.
5 . The neural network inference circuit of claim 1 , wherein the second block size is determined based on:
determining a first difference between a length of a first word and a length of other filters of the first word apart from a portion of the particular filter in the first word; and
subtracting the first difference from the first block size.
6 . The neural network inference circuit of claim 1 , wherein the input values for the particular layer are arranged in a plurality of two-dimensional grids, wherein each requested block of data comprising input values for the particular layer comprises at most one input value from each of the two-dimensional grids.
7 . The neural network inference circuit of claim 6 , wherein a particular requested block of data comprises one input value from each of a set of the two-dimensional grids, said input values in the particular requested block of data having a same set of coordinates in each respective two-dimensional grid of the set of two-dimensional grids.
8 . The neural network inference circuit of claim 7 , wherein the size of the particular requested block of data is based on a number of two-dimensional grids into which the input values for the particular layer are arranged.
9 . The neural network inference circuit of claim 1 , wherein the set of memories comprises random access memory (RAM), wherein the base memory address specifies a beginning of a particular fixed-length RAM word in the set of memories.
10 . The neural network inference circuit of claim 9 , wherein the location of the requested block of data specifies a number of blocks of data from the beginning of the particular fixed-length RAM word at which the requested block of data begins.
11 . The neural network inference circuit of claim 9 , wherein the read controller outputs to the first computing component, in response to each respective set of read instructions, a respective set of data having a fixed length of a RAM word, wherein the respective requested block of data is aligned to an edge of the respective output set of data.
12 . The neural network inference circuit of claim 11 , wherein when the size of the requested block of data is less than the fixed length of a RAM word, the output set of data comprises (i) the requested block of data and (ii) random data that is not used by the set of processing circuits.
13 . The neural network inference circuit of claim 1 , wherein the neural network inference circuit comprises a plurality of cores, each respective core comprising:
a respective subset of the processing circuits;
a respective subset of the memories; and
a respective read controller for retrieving data from the respective subset of memories for use by the respective subset of the processing circuits.
14 . The neural network inference circuit of claim 13 , wherein a subset of the cores are active for the particular layer, wherein each set of read instructions during execution of the particular layer is received by the respective read controller of each active core.
15 . The neural network inference circuit of claim 13 , wherein all of the memories belong to the plurality of cores, wherein an additional subset of the processing circuits used to execute the computation nodes of the neural network are separate from the plurality of cores.
16 . The neural network inference circuit of claim 9 , wherein the first block size is a fixed length of the RAM words.
17 . The neural network inference circuit of claim 1 , wherein the at least two different filters of the particular layer are stored as differently-sized blocks based on the different filters having different numbers of non-zero weight values.