Storage of input values within core of neural network inference circuit

1. A neural network inference circuit for executing a neural network that comprises a plurality of computation nodes at a plurality of layers, each of a set of the computation nodes comprising a dot product of input values and weight values, the neural network inference circuit comprising:

a set of dot product cores, each dot product core comprising:

a set of partial dot product computation circuits to compute dot products between sets of input values and sets of weight values; and

a set of memories to store weight values and input values for a particular layer of the neural network, wherein (i) the input values for the particular layer are arranged in a plurality of two-dimensional grids, (ii) a particular dot product core stores all of the input values of a subset of the two-dimensional grids, and (iii) input values having a same set of coordinates in each respective grid of the subset of the two-dimensional grids are stored sequentially within the set of memories of the particular dot product core; and

a set of dot product aggregation circuits for aggregating partial dot products from a plurality of the dot product cores.

2. The neural network inference circuit of claim 1 , wherein the input values for the particular layer are stored in a plurality of the dot product cores, wherein each dot product core of the plurality of dot product cores stores all of the input values of a different subset of the two-dimensional grids.

3. The neural network inference circuit of claim 2 , wherein each dot product core of the plurality of dot product cores stores the input values of a same number of the two-dimensional grids.

4. The neural network inference circuit of claim 2 , wherein the plurality of two-dimensional grids are divided between the plurality of dot product cores such that each dot product core of the plurality of dot product cores stores either (i) the input values of the same number of two-dimensional grids as the particular dot product core or (ii) the input values of one fewer two-dimensional grid than the particular dot product core.

5. The neural network inference circuit of claim 2 , wherein the plurality of two-dimensional grids are divided between the plurality of dot product cores such that each dot product core of the plurality of dot product cores stores either (i) the input values of the same number of two-dimensional grids as the particular dot product core or (ii) the input values of one more two-dimensional grid than the particular dot product core.

6. The neural network inference circuit of claim 1 , wherein each input value has (i) a two-dimensional coordinate pair indicating a row and column within the two-dimensional grid to which the input value belongs and (ii) a third coordinate indicating to which of the grids the input value belongs.

7. The neural network inference circuit of claim 6 , wherein the input values stored in the particular dot product core are arranged such that input values located in each row of their respective two-dimensional grids are stored in a contiguous block of the set of memories of the particular dot product core, with input values having the same coordinate pair in their respective two-dimensional grids stored in contiguous sub-blocks within the block for the row to which the coordinate pair belongs.

8. The neural network inference circuit of claim 7 , wherein:

the set of memories of each dot product core comprises a plurality of words;

each input value comprises a same number of bits such that each word has storage for a same number of input values; and

a first input value stored for a row is stored in a starting location of a particular word, with the input values located in the row stored in the particular word and subsequent words.

9. A neural network inference circuit for executing a neural network that comprises a plurality of computation nodes at a plurality of layers, each of a set of the computation nodes comprising a dot product of input values and weight values, the neural network inference circuit comprising:

a set of dot product cores, each dot product core comprising:

a set of partial dot product computation circuits to compute dot products between sets of input values and sets of weight values; and

a set of memories to store weight values and input values for a particular layer of the neural network, the set of memories comprising a plurality of words,

wherein:

the input values for the particular layer are arranged in a plurality of two-dimensional grids, each input value having (i) a two-dimensional coordinate pair indicating a row and column within the two-dimensional grid to which the input value belongs and (ii) a third coordinate indicating to which of the grids the input value belongs;

a particular dot product core stores all of the input values of a subset of the two-dimensional grids;

the input values stored in the particular dot product core are arranged such that input values located in each row of their respective two-dimensional grids are stored in a contiguous block of the set of memories of the particular dot product core, with input values having the same coordinate pair in their respective two-dimensional grids stored in contiguous sub-blocks within the block for the row to which the coordinate pair belongs;

each input value comprises a same number of bits such that each word has storage for a same number of input values;

a first input value stored for a row is stored in a starting location of a particular word, with the input values located in the row stored in the particular word and subsequent words; and

when a last input value stored for a row does not align with the ending location of a word, the remainder of the word in which the last input value for the row is stored is padded.

10. The neural network inference circuit of claim 9 , wherein the padding comprises random data that is not used for any of the computation nodes of the layer.

11. The neural network inference circuit of claim 1 , wherein:

the particular dot product core is a first dot product core of the plurality of dot product cores and the subset of the two-dimensional grids is a first subset of the two-dimensional grids;

a second dot product core of the plurality of dot product cores stores all of the input values of a second subset of the two-dimensional grids; and

input values having a same set of coordinates in each grid of the second subset of the two-dimensional grids are stored sequentially within the set of memories of the second dot product core.

12. The neural network inference circuit of claim 1 , wherein:

the set of memories of each dot product core comprises a plurality of words; and

each input value comprises a same number of bits such that each word has storage for a same number of input values.

13. A neural network inference circuit for executing a neural network that comprises a plurality of computation nodes at a plurality of layers, each of a set of the computation nodes comprising a dot product of input values and weight values, the neural network inference circuit comprising:

a set of dot product cores, each dot product core comprising:

a set of partial dot product computation circuits to compute dot products between sets of input values and sets of weight values; and

a set of memories to store weight values and input values for a particular layer of the neural network, the set of memories comprising a plurality of words,

wherein:

the input values for the particular layer are arranged in a plurality of two-dimensional grids;

a particular dot product core stores all of the input values of a subset of the two-dimensional grids;

input values having a same set of coordinates in each respective grid of the subset of the two-dimensional grids are stored sequentially within the set of memories of the particular dot product core;

each input value comprises a same number of bits such that each word has storage for a same number of input values; and

when the number of two-dimensional grids in the subset of two-dimensional grids stored by the particular dot product core is larger than the number of input values stored in each word:

the subset of two-dimensional grids is divided into blocks having the same number of two-dimensional grids as the number of input values stored in each word; and

the input values having a same set of coordinates in each respective grid of a first block are stored sequentially within a first word of the set of memories of the particular dot product core while input values having the same set of coordinates in each respective grid of a second block are stored sequentially within a second word of the set of memories of the particular dot product core, wherein the first and second words are not sequential within the set of memories of the particular dot product core.

14. The neural network inference circuit of claim 13 , wherein each word is a 128-bit random access memory (RAM) word, wherein each input value is a 4-bit value.

15. The neural network inference circuit of claim 1 , wherein a particular computation node of the particular layer uses a subset of the input values stored in the particular dot product core.

16. The neural network inference circuit of claim 15 , wherein the subset of the input values comprises all of the input values stored in the particular dot product core having a particular subset of coordinate sets in each respective two-dimensional grid of the plurality of two-dimensional grids.

17. The neural network inference circuit of claim 16 , wherein the particular subset of coordinate sets are contiguous within each respective two-dimensional grid of the plurality of two-dimensional grids.

18. The neural network inference circuit of claim 13 further comprising a set of dot product aggregation circuits for aggregating partial dot products from a plurality of the cores.

19. The neural network inference circuit of claim 1 , wherein:

the particular layer is a first layer and the plurality of two-dimensional grids is a first plurality of two-dimensional grids;

the input values for a second layer are arranged in a second plurality of two-dimensional grids;

the particular dot product core stores all of the input values of a subset of the second plurality of two-dimensional grids; and

input values having a same set of coordinates in each respective grid of the subset of the second plurality of two-dimensional grids are stored sequentially within the set of memories of the particular dot product core.

20. The neural network inference circuit of claim 19 , wherein the input values of the second layer are output values of the first layer.