On-chip interconnect for memory channel controllers

1. An integrated circuit configured to accelerate machine-learning computations, the circuit comprising:

a plurality of processor cores, each processor core comprising:

a plurality of channel controllers;

an interface controller configured to couple each of the plurality of channel controllers to each memory channel of a system memory; and

a respective fetch unit in each of the plurality of channel controllers, the respective fetch unit being configured to:

i) receive a request that encodes addressing information;

ii) obtain, based on the addressing information, data from any memory channel of the system memory using the interface controller; and

iii) write the data obtained from any memory channel to a vector memory of the processor core via the channel controller that includes the fetch unit.

2. The integrated circuit of claim 1 , wherein the interface controller is operable to control:

an intra-memory OCI node that includes:

i) a plurality of read interfaces that are used to obtain data from any memory location along any memory channel of the system memory; and

ii) a plurality of write interfaces that are used to write data to any memory location along any memory channel of the system memory.

3. The integrated circuit of claim 2 , wherein the intra-memory OCI node comprises:

a first in, first out (FIFO) memory queue configured to store incoming OCI transaction requests during arbitration of an existing OCI transaction for: i) obtaining data from any memory location of the system memory or ii) writing data to any memory location of the system memory.

4. The integrated circuit of claim 3 , wherein the interface controller is configured to:

receive the request that encodes the addressing information;

generate a plurality of control signals based on a plurality of IDs in the addressing information; and

provide the plurality of control signals to any memory channel of the system memory to fetch a data stored in a memory location of the system memory.

5. The integrated circuit of claim 4 , wherein:

the addressing information is derived from one or more incoming OCI transaction requests received at the interface controller; and

at least one existing OCI transaction request for accessing any memory location of the system memory was previously stored in the FIFO memory queue.

6. The integrated circuit of claim 5 , wherein:

each incoming OCI transaction request comprises a plurality of identifier's (IDs) corresponding to any set of memory locations across any memory channel of the system memory.

7. The integrated circuit of claim 6 , wherein:

the OCI interface includes two or more addressing modes; and

the interface controller is operable to generate control signals that allow for selecting between the two or more addressing modes of the interface controller.

8. The integrated circuit of claim 7 , wherein:

a first addressing mode of the two or more addressing modes is a channel-interleaved mode that constrains a mapping of an embedding table relative to memory channels of the system memory.

9. The integrated circuit of claim 8 , wherein:

a second addressing mode of the two or more addressing modes is a stack-interleaved mode that expands the mapping of the embedding table relative to memory channels of the system memory.

10. The integrated circuit of claim 7 , wherein each OCI transaction request is:

received by the fetch unit; and

processed, by the interface controller, to initiate accessing any memory location indicated in the OCI transaction request.

11. The integrated circuit of claim 10 , wherein each OCI transaction request for accessing data stored in the system memory encodes:

i) a length in 32 Bytes corresponding to the data being accessed;

ii) a TokenID that represents a source address specific to any memory location in system memory; and

iii) a destination address of a memory location of the vector memory.

12. The integrated circuit of claim 8 , wherein:

the embedding table is split-up as multiple data shards that are allocated across memory channels of the entire system memory; and

the interface controller selects the stack-interleaved mode to access any portion of the embedding table using any one of the plurality of channel controllers.

13. The integrated circuit of claim 1 , further comprising:

a respective flush unit in each of the plurality of channel controllers, the respective flush unit being configured to:

i) receive a request that encodes addressing information;

ii) write, based on the addressing information, data from a source address in the vector memory to any memory location of the system memory using the interface controller.

14. The integrated circuit of claim 13 , wherein:

the machine-learning computations are performed to process neural network inputs through a neural network layer; and

the flush unit is used to write parameters for the neural network layer to any memory location of the system memory using the interface controller.

15. A computer-implemented method performed using an integrated circuit configured to accelerate machine-learning computations, the integrated circuit comprising a system memory and a processor core that includes a plurality of channel controllers, the method comprising:

receiving a plurality of requests, where each of the plurality of requests encodes addressing information;

for a first request of the plurality of requests:

identifying, based on addressing information of the first request, any memory location of any memory channel of the system memory;

obtaining data from a memory location among any memory location of the system memory using an on-chip interconnect (OCI) interface that couples each of the plurality of channel controllers to each memory channel of the system memory; and

writing, based on the addressing information of the first request, the data obtained from the memory location to a vector memory of the processor core.

16. The method of claim 15 , wherein:

the OCI interface comprises an intra-memory OCI node that includes a plurality of read interfaces and a plurality of write interfaces; and

the method further comprises:

obtaining, based on control signals generated by an interface controller, data from any memory location along any memory channel of the system memory using the plurality of read interfaces; and

writing, based on control signals generated by the interface controller, data to any memory location along any memory channel of the system memory using the plurality of write interfaces.

17. The method of claim 16 , wherein the intra-memory OCI node comprises a first in, first out (FIFO) memory queue and the method comprises:

storing, at the FIFO memory queue, incoming OCI transaction requests during arbitration of an existing OCI transaction for: i) obtaining data from any memory location of the system memory or ii) writing data to any memory location of the system memory.

18. The method of claim 17 , further comprising:

receiving, by the interface controller, the request that encodes the addressing information;

generating, by the interface controller, a plurality of control signals based on a plurality of IDs in the addressing information; and

providing, by the interface controller, the plurality of control signals to any memory channel of the system memory to fetch a data stored in a memory location of the system memory.

19. The method of claim 18 , wherein:

the addressing information is derived from one or more incoming OCI transaction requests received at the interface controller; and

at least one existing OCI transaction request for accessing any memory location of the system memory was previously stored in the FIFO memory queue.

20. The method of claim 19 , wherein:

each incoming OCI transaction request comprises a plurality of identifier's (IDs) corresponding to any set of memory locations across any memory channel of the system memory.