Data layout conscious processing in memory architecture for executing neural network model
The present disclosure relates to a processing in memory (PIM) enabled device for executing a neural network model. The PIM enabled device comprises a memory block assembly comprising a first array of memory blocks, a second array of memory blocks adjacent to the first array of memory blocks, a plurality of first data links associated with the first array of memory blocks and the second array of memory blocks, wherein each data link of the plurality of first data links communicatively couples two corresponding memory blocks of which are from the first array of memory blocks and the second array of memory blocks respectively, and a second data link communicatively coupled to the plurality of first data links. The data from a first memory block of the first array of memory blocks can be transferable to a second memory block of the second array of memory blocks via the plurality of first data links and the second data link.
1. A processing in memory (PIM) enabled device for executing a neural network model, comprising:
a memory block assembly with a plurality of rows, comprising:
a first array of memory blocks arranged within the plurality of rows;
a second array of memory blocks arranged within the plurality of rows adjacent to the first array of memory blocks;
a plurality of first data links associated with the first array of memory blocks and the second array of memory blocks, wherein each data link of the plurality of first data links communicatively couples two corresponding memory blocks of a same row of the plurality of rows from the first array of memory blocks and the second array of memory blocks respectively; and
a second data link comprising two parallel column data links arranged between the first array of memory blocks and the second array of memory blocks and directly communicatively coupled to the plurality of first data links, respectively,
wherein data from a first memory block of a first row of the first array of memory blocks is transferable to a second memory block of a second row different from the first row of the second array of memory blocks via the plurality of first data links and either or both of the two column data links of the second data link.
2. The PIM enabled device of claim 1 , wherein each memory block includes a memory cell array arranged in a first plurality of rows and in a second plurality of columns.
3. The PIM enabled device of claim 1 , wherein the plurality of first data links and the second data link are configured to transfer the data stored in a column of memory cells in the first memory block to a row of memory cells in the second memory block in one cycle.
4. The PIM enabled device of claim 1 , wherein the memory block assembly further comprises:
a third array of memory blocks arranged within the plurality of rows adjacent to the second array of memory blocks;
a plurality of third data links associated with the second array of memory blocks and the third array of memory blocks, wherein each of data link of the plurality of third data links communicatively couples two corresponding memory blocks of a same row of the plurality of rows from the second array of memory blocks and the third array of memory blocks respectively;
a fourth data link communicatively coupled to the plurality of third data links; and
a transfer buffer communicatively coupled with the second data link and the fourth data link,
wherein data from the first memory block of a first row of the first array of memory blocks is transferable to a third memory block of a second row different from the first row of the third array of memory blocks via the transfer buffer.
5. The PIM enabled device of claim 4 , wherein the memory block assembly further comprises:
a first buffer data link configured to communicatively couple the second data link and the transfer buffer; and
a second buffer data link configured to communicatively couple the fourth data link and the transfer buffer.
6. The PIM enabled device of claim 5 , wherein the memory block assembly further comprises:
a first switch configured to control opening or closing a data path from the first memory block to a first data link communicatively coupled with the first memory block among the plurality of first data links;
a second switch configured to control opening or closing a data path from the first buffer data link to the transfer buffer;
a third switch configured to control opening or closing a data path from the transfer buffer to the second buffer data link; and
a fourth switch configured to control opening or closing a data path from a third data link communicatively coupled with the third memory block among the plurality of third data links to the third memory block.
7. The PIM enabled device of claim 6 , wherein the memory block assembly further comprises:
a first selector configured to select the first switch to open the data path from the first memory block to the first data link communicatively coupled with the first memory block;
a second selector configured to select the second switch to open the data path from the first buffer data link to the transfer buffer;
a third selector configured to select the third switch to open the data path from the transfer buffer to the second buffer data link and
a fourth selector configured to select the fourth switch to open the data path from the third data link communicatively coupled with the third memory block to the third memory block.
8. The PIM enabled device of claim 4 , wherein each memory block includes a memory cell array arranged in a first plurality of rows and in a second plurality of columns.
9. The PIM enabled device of claim 4 , wherein the plurality of first data links, the second data link, and the first buffer data link are configured to transfer the data stored in a column of memory cells in the first memory block to a row of memory cells in the transfer buffer in one cycle.
10. The PIM enabled device of claim 9 , wherein the second buffer data link, the fourth data link, and the plurality of third data links are configured to transfer the data stored in the row of memory cells in the transfer buffer to a row of memory cells in the third memory block in another cycle.
11. The PIM enabled device of claim 1 , further comprising a block table for recording a status for each memory block in the memory block assembly,
wherein the status includes computing status indicating a corresponding memory block is executing an assigned operation, ready status indicating a corresponding memory block has data ready for being transferred, and an idle status indicating a corresponding memory is ready for receiving data.
12. The PIM enabled device of claim 1 , further comprising a data transfer table for recording pending data transfer in the memory block assembly,
wherein the data transfer table includes a source field indicating a memory block sending data and a destination field indicating a memory block receiving the data.
13. The PIM enabled device of claim 12 , wherein the data transfer table further includes a buffer field indicating a buffer the data resides in.
14. The PIM enabled device of claim 1 , further comprising a controller configured to provide commands to the memory block assembly to execute a first operation of the neural network model on the first memory block and a second operation of the neural network model on the second memory block, the second operation uses output data from execution of the first operation as an input.
15. The PIM enabled device of claim 1 , wherein the memory block assembly further comprises:
a first switch configured to control opening or closing a data path from the first memory block to a first data link communicatively coupled with the first memory block among the plurality of first data links; and
a second switch configured to control opening or closing a data path from a first data link communicatively coupled with the second memory block among the plurality of first data links to the second memory block.
16. A method for executing a neural network model on a processing in memory (PIM) enabled device comprising a memory block assembly, comprising:
processing a first operation of the neural network model on a first memory block of a first array of memory blocks in the memory block assembly comprising the first array of memory blocks and a second array of memory blocks that are arranged within a plurality of rows;
transferring output data from the first memory block of a first row of the plurality of rows to a second memory block of a second row of the plurality of rows of the second array of memory blocks via data links communicatively coupling the first memory block and the second memory in the memory block assembly depending on the first row and the second row, wherein the data links comprises a plurality of first data links and a second data link communicatively coupled to the plurality of first data links, the second data link comprising two parallel column data links arranged between the first array of memory blocks and the second array of memory blocks and directly communicatively coupled to the plurality of first data links, respectively, and transferring output data is performed in one or more cycles depending on a size of the output data and available memory in source and destination block; and
processing, on the second memory block, a second operation of the neural network model based on the output data.
17. The method of claim 16 , further comprising:
transferring the output data from the first memory block to a third memory block of the second array of memory blocks; and
processing, on the third memory block, a third operation of the neural network model based on the output data,
wherein transferring the output data from the first memory block to the third memory block is performed after or during a time period of transferring the output data from the first memory block to the second memory block.
18. The method of claim 16 , wherein transferring output data comprises:
transferring the output data from the first memory block to a transfer buffer in the memory block assembly; and
transferring the output data from the transfer buffer to the second memory block,
wherein the transfer buffer is communicatively coupled with the first memory block and the second memory block.
19. The method of claim 16 , wherein transferring output data comprises:
transferring the output data stored in a column of memory cells in the first memory block to a row of memory cells in the second memory block.
20. The method of claim 16 , further comprising:
recording a status for each memory block in the memory block assembly in a block table,
wherein the status includes a computing status indicating a corresponding memory block is executing an assigned operation, a ready status indicating a corresponding memory block has data ready for being transferred, and an idle status indicating a corresponding memory is ready for receiving data.
21. The method of claim 16 , further comprising:
recording pending data transfer in the memory block assembly in a data transfer table,
wherein the data transfer table includes a source field indicating a memory block sending data and a destination field indicating a memory block receiving the data.
22. A non-transitory computer readable storage media storing a set of instructions that are executable by at least one processor of a processing in memory (PIM) enabled terminal to cause the terminal to perform a method comprising:
processing a first operation of the neural network model on a first memory block of a first array of memory blocks in a memory block assembly comprising the first array of memory blocks and a second array of memory blocks that are arranged within a plurality of rows;
transferring output data from the first memory block of a first row of the plurality of rows to a second memory block of a second row of the plurality of rows of the second array of memory blocks via data links communicatively coupling the first memory block and the second memory in the memory block assembly depending on the first row and the second row, wherein the data links comprises a plurality of first data links and a second data link communicatively coupled to the plurality of first data links, the second data link comprising two parallel column data links arranged between the first array of memory blocks and the second array of memory blocks and communicatively coupled to the plurality of first data links, respectively, and transferring output data is performed in one or more cycles depending on a size of the output data and available memory in source and destination block; and
processing, on the second memory block, a second operation of the neural network model based on the output data.
23. The computer readable medium of claim 22 , wherein the set of instructions that is executable by at least one processor of the terminal to cause the terminal to further perform:
transferring the output data from the first memory block to a third memory block of the second array of memory blocks; and
processing, on the third memory block, a third operation of the neural network model based on the output data,
wherein transferring the output data from the first memory block to the third memory block is performed after or during a time period of transferring the output data from the first memory block to the second memory block.
24. The computer readable medium of claim 22 , wherein the set of instructions that is executable by at least one processor of the terminal to cause the terminal to further perform:
transferring the output data from the first memory block to a transfer buffer in the memory block assembly; and
transferring the output data from the transfer buffer to the second memory block,
wherein the transfer buffer is communicatively coupled with the first memory block and the second memory block.
25. The computer readable medium of claim 22 , wherein transferring output data comprises:
transferring the output data stored in a column of memory cells in the first memory block to a row of memory cells in the second memory block.
26. The computer readable medium of claim 22 , wherein the set of instructions that is executable by at least one processor of the terminal to cause the terminal to further perform:
recording a status for each memory block in the memory block assembly in a block table,
wherein the status includes a computing status indicating a corresponding memory block is executing an assigned operation, a ready status indicating a corresponding memory block has data ready for being transferred, and an idle status indicating a corresponding memory is ready for receiving data.
27. The computer readable medium of claim 22 , wherein the set of instructions that is executable by at least one processor of the terminal to cause the terminal to further perform:
recording pending data transfer in the memory block assembly in a data transfer table,
wherein the data transfer table includes a source field indicating a memory block sending data and a destination field indicating a memory block receiving the data.
28. A terminal, comprising:
a host unit; and
a processing in memory (PIM) enabled device communicatively coupled to the host unit, the PIM enabled device comprising:
a memory block assembly with a plurality of rows, comprising:
a first array of memory blocks arranged within the plurality of rows;
a second array of memory blocks arranged within the plurality of rows adjacent to the first array of memory blocks;
a plurality of first data links associated with the first array of memory blocks and the second array of memory blocks, wherein each data link of the plurality of first data links communicatively couples two corresponding memory blocks of a same row of the plurality of rows from the first array of memory blocks and the second array of memory blocks respectively; and
a second data link communicatively coupled to the plurality of first data links, the second data link comprising two parallel column data links arranged between the first array of memory blocks and the second array of memory blocks and directly communicatively coupled to the plurality of first data links, respectively,
wherein data from a first memory block of a first row of the first array of memory blocks is transferable to a second memory block of a second row different from the first row of the second array of memory blocks via the plurality of first data links and either or both of the two column data links of the second data link.
29. The PIM enabled device of claim 28 , wherein the memory block assembly further comprises:
a first selector configured to select the first switch to open the data path from the first memory block to the first data link communicatively coupled with the first memory block; and
a second selector configured to select the second switch to open the data path from the first data link communicatively coupled with the second memory block to the second memory block.