Processor and memory communication in a stacked memory system
Embodiments of the present disclosure relate to application partitioning for locality in a stacked memory system. In an embodiment, one or more memory dies are stacked on the processor die. The processor die includes multiple processing tiles and each memory die includes multiple memory tiles. Vertically aligned memory tiles are directly coupled to and comprise the local memory block for a corresponding processing tile. An application program that operates on dense multi-dimensional arrays (matrices) may partition the dense arrays into sub-arrays associated with program tiles. Each program tile is executed by a processing tile using the processing tile's local memory block to process the associated sub-array. Data associated with each sub-array is stored in a local memory block and the processing tile corresponding to the local memory block executes the program tile to process the sub-array data.
1 . A method, comprising:
generating a memory access request by a first processing unit within a first processing tile of a two dimensional (2D) array of processing tiles that are fabricated within a processor die, wherein the processor die and at least one memory die are stacked in a third dimension;
transmitting the memory access request comprising an address to a first mapper within the first processing tile over a narrow sub-network within the first processing tile, wherein the narrow sub-network transmits read requests and write replies;
transmitting data associated with the memory access request from the first processing unit directly to a first tile network within the first processing tile over a wide sub-network, wherein the wide sub-network bypasses the first mapper and transmits write requests and read replies;
translating the address by the first mapper to form a translated memory address request that is transmitted through the narrow sub-network to the first tile network within the first processing tile; and
transmitting the translated memory access request including the data from the first processing tile to a first memory tile of a plurality of memory tiles that are fabricated within the at least one memory die through first conductive paths of a plurality of conductive paths that couple each processing tile in the 2D array to a corresponding one of the memory tiles in each memory die of the at least one memory die for communication between each processing tile and the corresponding memory tile, wherein the corresponding memory tile is stacked on the processing tile in the third dimension.
2 . The method of claim 1 , wherein the processor die and the at least one memory die are enclosed within an integrated circuit package.
3 . The method of claim 1 , wherein the processor die comprises a graphics processing unit.
4 . The method of claim 1 , wherein the processor die comprises at least one central processing unit.
5 . The method of claim 1 , wherein the at least one memory die comprises a first memory die that is disposed between the processor die and a second memory die.
6 . The method of claim 1 , further comprising a tile communication network for transmitting translated memory access requests from each processing tile in the 2D array to memory tiles coupled to different processing tiles in the 2D array, wherein the tile communication network is fabricated in the processor die and connects each processing tile with adjacent processing tiles in a first dimension of the 2D array and with adjacent processing tiles in a second dimension of the 2D array.
7 . The method of claim 6 , wherein I/O circuitry for the tile communication network is fabricated at a perimeter of each processing tile in the 2D array.
8 . The method of claim 6 , wherein a bandwidth capacity of the tile communication network is less than or equal to a memory bandwidth capacity of the first conductive paths.
9 . The method of claim 1 , wherein the conductive paths comprise a through-die via structure that is fabricated within the at least one memory die.
10 . The method of claim 1 , wherein at least one of the steps of generating, transmitting the memory access request, transmitting data, translating, and transmitting the translated memory access request is performed on a server or in a data center to generate an image, and the image is streamed to a user device.
11 . The method of claim 1 , wherein at least one of the steps of generating, transmitting the memory access request, transmitting data, translating, and transmitting the translated memory access request is performed within a cloud computing environment.
12 . The method of claim 1 , wherein at least one of the steps of generating, transmitting the memory access request, transmitting data, translating, and transmitting the translated memory access request is performed for training, testing, or inferencing with a neural network employed in a machine, robot, or autonomous vehicle.
13 . The method of claim 1 , wherein, based on the address, the first mapper performs a predetermined operation on bits of the address to form the translated address comprising an identifier corresponding to the first memory tile and an offset location within the first memory tile.
14 . The method of claim 1 , wherein, based on the address, the first mapper obtains a segment descriptor comprising an identifier corresponding to the first memory tile and fields in the segment descriptoraddress that specify widths of bit fields used to determine a starting location within the first memory tile.
15 . The method of claim 1 , wherein the memory request comprises an atomic memory operation and the wide sub-network transmits the data for invoking execution of a processing thread using the data.
16 . A system, comprising:
at least one device, each device comprising:
at least one stack of dies, each stack of dies including a processor die comprising a first processing tile in a two dimensional (2D) array of processing tiles that are fabricated within a processor die;
at least one memory die, each memory die comprising a plurality of memory tiles, wherein the processor die and the at least one memory die are stacked in a third dimension; and
conductive paths between each processing tile and a corresponding memory tile in each memory die of the at least one memory die, wherein
a memory access request comprising an address is generated by a first processing unit within a first processing tile in the 2D array,
the memory access request is transmitted to a first mapper within the first processing tile over a narrow sub-network within the first processing tile, wherein the narrow sub-network transmits read requests and write replies,
data associated with the memory access request is transmitted from the first processing unit directly to a first tile network within the first processing tile over a wide sub-network, wherein the wide sub-network bypasses the first mapper and transmits write requests and read replies,
the address is translated by the first mapper to form a translated memory address request that is transmitted through the narrow sub-network to the first tile network within the first processing tile, and
the translated memory access request including the data is transmitted by the first processing tile to a first memory tile of a plurality of memory tiles through first conductive paths of a plurality of conductive paths that couple each processing tile in the 2D array to a corresponding one of the memory tiles in each memory die of the at least one memory die for communication between each processing tile and the corresponding memory tile, wherein the corresponding memory tile is stacked on the processing tile in the third dimension.
17 . The system of claim 16 , wherein the conductive paths comprise a through-die via structure in the third dimension that is fabricated within each one of the at least one memory die.
18 . The system of claim 16 , further comprising a tile communication network for transmitting translated memory access requests from each processing tile in the 2D array to memory tiles coupled to different processing tiles in the 2D array, wherein the tile communication network is fabricated in the processor die and connects each processing tile with adjacent processing tiles in a first dimension of the 2D array and with adjacent processing tiles in a second dimension of the 2D array.
19 . The system of claim 18 , wherein I/O circuitry for the tile communication network is fabricated at a perimeter of each processing tile in the 2D array.
20 . The system of claim 18 , wherein a bandwidth capacity of the tile communication network is less than or equal to a memory bandwidth capacity of the first conductive paths.