Tiled in-memory computing architecture
A compute tile is described. The compute tile includes compute engines and a general-purpose (GP processor coupled with the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module is configured to store weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix. The GP processor is configured to control the compute engines, to receive output of the VMM for the matrix from the compute engines, and to perform a nonlinear operation on the output. The compute engines are addressable by data movement initiators. Data may be moved to and/or from the compute engines in data paths that bypass the GP processor.
1 . A compute tile, comprising:
a plurality of compute engines, each of the plurality of compute engines including a compute-in-memory (CIM) hardware module, the CIM hardware module being configured to store a plurality of weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix;
a general-purpose (GP) processor coupled with the plurality of compute engines and configured to control the plurality of compute engines, to receive output of the VMM for the matrix from each of the plurality of compute engines, and to perform a nonlinear operation on the output, the GP processor including a fixed function computing block usable in performing the nonlinear operation, the nonlinear operation corresponding to an activation function for the output of the VMM; and
a data conversion engine coupled with the plurality of compute engines and configured to convert data transferred to the plurality of compute engines from a first format to a second format, wherein the data was received from another compute tile in the first format;
wherein the plurality of compute engines is addressable by a plurality of data movement initiators and by the GP processor.
2 . The compute tile of claim 1 , wherein the plurality of compute engines and the plurality of data movement initiators are configured to at least one of move data to the plurality of compute engines in a data path bypassing the GP processor or move data from the plurality of compute engines in a data path bypassing the GP processor of the compute tile.
3 . The compute tile of claim 1 , further comprising:
a direct memory access (DMA) unit, the plurality of data movement initiators including at least one of the DMA unit or a component on an additional compute tile coupled with the compute tile.
4 . The compute tile of claim 3 , further comprising:
a local memory coupled with the plurality of compute engines and the GP processor, the DMA unit being configured to transfer data between the local memory and the plurality of compute engines in a data path bypassing the GP processor.
5 . The compute tile of claim 1 , wherein the data conversion engine includes at least one of a BFloat-Integer format converter or a reshape engine, the reshape engine configured to pad the data.
6 . The compute tile of claim 1 , further comprising:
a local memory coupled with the plurality of compute engines through a first bus, the GP processor being coupled with the plurality of compute engines through a second bus different from the first bus; and
a main bus coupled with the local memory and the GP processor.
7 . The compute tile of claim 1 , further comprising:
a local memory coupled with the plurality of compute engines by a first bus; and
a main bus coupled with the local memory, the plurality of compute engines, and the GP processor, the GP processor being coupled with the plurality of compute engines and the local memory through the main bus.
8 . The compute tile of claim 7 , wherein the plurality of compute engines is coupled to the main bus through an interconnect having an internal queueing.
9 . The compute tile of claim 1 , wherein the compute tile further includes:
a local compute engine memory having a first memory density; and
a cache controller coupled with the local compute engine memory;
wherein the CIM hardware module is coupled with the cache controller and has a second memory density less than the first memory density.
10 . A system, comprising:
a plurality of compute tiles, each of the plurality of compute tiles including a plurality of compute engines and a general-purpose (GP) processor, each of the plurality of compute engines including a compute-in-memory (CIM) hardware module, the CIM hardware module being configured to store a plurality of weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix, the GP processor being coupled with the plurality of compute engines and being configured to control the plurality of compute engines, to receive output of the VMM for the matrix from each of the plurality of compute engines, and to perform a nonlinear operation on the output, the GP processor including a fixed function computing block usable in performing the nonlinear operation, the plurality of compute tiles including a first compute tile and a second compute tile, the nonlinear operation corresponding to an activation function for the VMM; and
a data conversion engine coupled with the plurality of compute engines of the first compute tile and configured to convert data transferred to the plurality of compute engines from a first format to a second format, wherein the data was received from the second compute tile in the first format;
wherein the plurality of compute engines is addressable by a plurality of data movement initiators on the plurality of compute tiles and by the GP processor, the plurality of data movement initiators being configured to move data to the plurality of compute engines on a the first compute tile of the plurality of compute tiles in a data path bypassing the GP processor on the first compute tile.
11 . A method, comprising:
providing an input vector to at least one compute engine, the at least one compute engine being part of a plurality of compute engines on a first compute tile of a plurality of compute tiles, the plurality of compute engines being coupled with a general-purpose (GP) processor on the first compute tile, the at least one compute engine storing a plurality of weights corresponding to a matrix, each of the plurality of compute engines including a compute-in-memory (CIM) hardware module, the CIM hardware module of the at least one compute engine being configured to store the plurality of weights in a plurality of storage cells and to perform a vector-matrix multiplication (VMM) of the matrix and the input vector, the at least one compute engine performing the VMM for the input vector and the matrix to provide an output, the CIM hardware module configured to perform the VMM by a bit-wise multiplication of the plurality of weights stored in the plurality of storage cells of the CIM hardware module and elements of the input vector;
applying, by the GP processor, a function to the output, the GP processor being configured to control the plurality of compute engines, the GP processor including a fixed function computing block usable in performing the function, the function being a nonlinear activation function for the VMM; and
converting data transferred to the plurality of compute engines of the first compute tile from a first format to a second format, wherein the data was received from a second compute tile of the plurality of compute tiles in the first format;
wherein the plurality of compute engines is addressable by a plurality of data movement initiators and by the GP processor such that the providing the input vector includes providing the input vector to the at least one compute engine by a data movement initiator of the plurality of data movement initiators in a data path that bypasses the GP processor or providing the input vector to the at least one compute engine by the GP processor.
12 . The method of claim 11 , further comprising:
moving data from the CIM hardware module to the at least one compute engine;
wherein the data is moved from the CIM hardware module to the at least one compute engine via at least one data path bypassing the GP processor.
13 . The method of claim 11 , wherein the providing the input vector further includes:
using at least one of a direct memory access (DMA) unit, the plurality of data movement initiators including the DMA unit, or a component on a third compute tile coupled with the first compute tile.
14 . The method of claim 11 , wherein a local memory is coupled with the plurality of compute engines through a first bus, the GP processor is coupled with the plurality of compute engines through a second bus different from the first bus, and a main bus is coupled with the local memory and the GP processor.
15 . The method of claim 11 , wherein a local memory is coupled with the plurality of compute engines by a first bus and wherein a main bus is coupled with the local memory, the plurality of compute engines, and the GP processor, the GP processor being coupled with the plurality of compute engines and the local memory through the main bus.
16 . The method of claim 15 , wherein the plurality of compute engines is coupled to the main bus through an interconnect having an internal queueing.
17 . The method of claim 11 , wherein each of the plurality of compute tiles further includes a local compute engine memory having a first memory density and a cache controller coupled with the local compute engine memory, and wherein the CIM hardware module is coupled with the cache controller and has a second memory density less than the first memory density.
18 . The compute tile of claim 1 , wherein the GP processor is reduced instruction set computer (RISC) processor configured as a separate data movement initiator providing data to the plurality of compute engines, the data including a vector for the VMM.
19 . The compute tile of claim 1 , wherein each of the plurality of compute engines includes a local update module configured to update at least one weight of the plurality of weights in the CIM hardware module.
20 . The compute tile of claim 1 , further comprising:
a local memory coupled with the GP processor;
wherein each compute engine of the plurality of compute engines is coupled with a local compute engine memory, the local compute engine memory having a first memory density;
wherein each compute engine of the plurality of compute engines is coupled to a cache controller, the cache controller being coupled with the local compute engine memory;
wherein the CIM hardware module of each compute engine is coupled with the cache controller and has a second memory density less than the first memory density.
21 . The compute tile of claim 1 , wherein the GP processor is a single GP processor for the compute tile, the single GP processor controlling the plurality of compute engines.