Systems and methods for data placement for in-memory-compute
According to one embodiment, a memory module includes: a memory die including a dynamic random access memory (DRAM) banks, each including: an array of DRAM cells arranged in pages; a row buffer to store values of one of the pages; an input/output (IO) module; and an in-memory compute (IMC) module including: an arithmetic logic unit (ALU) to receive operands from the row buffer or the IO module and to compute an output based on the operands and one of a plurality of ALU operations; and a result register to store the output of the ALU; and a controller to: receive, from a host processor, operands and an instruction; determine, based on the instruction, a data layout; supply the operands to the DRAM banks in accordance with the data layout; and control an IMC module to perform one of the ALU operations on the operands in accordance with the instruction.
1. A memory module comprising:
a dynamic random access memory (DRAM) device comprising:
an array of DRAM rows; and
an in-memory compute (IMC) module; and
a memory controller configured to:
receive, from a host processor, an input data and an instruction; and
supply the input data to the DRAM device in a data arrangement selected based on the instruction, the data arrangement specifying placement of an operand among the array of DRAM rows and the IMC module,
wherein the IMC module is configured to perform an operation on the input data based on the instruction.
2. The memory module of claim 1 , wherein the data arrangement specifies:
a first input data is read from the array of DRAM rows and the input data comprises a second input data, the second input data being supplied from the host processor to the IMC module.
3. The memory module of claim 2 , wherein the IMC module comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of the first input data in the input buffer; and
perform the operation on the first input data stored in the input buffer and second portions of the second input data.
4. The memory module of claim 1 , wherein the input data is divided into first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store one or more of the first portions and one or more of the second portions in a row of the array of DRAM rows.
5. The memory module of claim 1 , wherein the IMC module comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of first portions of the input data in the input buffer, the first portions being stored in a first row of the array of DRAM rows; and
perform the operation on the first portion stored in the input buffer and second portions of the input data stored in the first row of the array of DRAM rows.
6. The memory module of claim 1 , wherein the IMC module comprises an accumulator configured to store an accumulated value, the accumulator being configured to:
receive an output of the operation; and
update the accumulator with a sum of the accumulated value and the output.
7. The memory module of claim 1 , wherein a first portion of the input data has a first number of values and a second portion of the input data has a second number of values,
wherein the IMC module comprises an output buffer configured to store an outer product of the first portion and the second portion.
8. The memory module of claim 1 , wherein a first portion of the input data has a first number of values and a second portion of the input data has a second number of values,
wherein the IMC module comprises an output buffer configured to store a tensor product of the first portion and the second portion.
9. The memory module of claim 1 , wherein the input data comprises first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store a subset of the first portions in a first row of the array of DRAM rows and a subset of the second portions in a second row of the array of DRAM rows.
10. The memory module of claim 1 , wherein the IMC module further comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of input data from a first row of the array of DRAM rows in the input buffer; and
perform the operation on the first portion stored in the input buffer and second portions of the input data stored in a second row of the array of DRAM rows.
11. The memory module of claim 1 , wherein the IMC module comprises a hardware buffer configured to buffer an output of the operation.
12. The memory module of claim 1 , wherein the IMC module comprises an accumulator configured to store an accumulated value, the accumulator being configured to:
receive an output of the operation, the operation comprising an inner product operation; and
update the accumulator with a sum of the accumulated value and the output.
13. The memory module of claim 1 , wherein the memory module is a high bandwidth memory (HBM) module comprising a stack of memory dies connected by through silicon vias.
14. A method for performing computations in-memory comprising:
receiving, by a memory controller of a memory module, an input data and an instruction;
determining, by the memory controller, based on the instruction, a data arrangement;
supplying the input data to a dynamic random access memory (DRAM) device in the data arrangement, the DRAM device comprising:
an array of DRAM rows; and
an in-memory compute (IMC) module,
wherein the IMC module is configured to perform an operation on the input data based on the instruction,
wherein the data arrangement specifies placement of the input data in the array of DRAM rows and the IMC module.
15. The method of claim 14 , wherein the data arrangement specifies:
a first input data is read from the DRAM rows and the input data comprises a second input data supplied from a host processor to the IMC module.
16. The method of claim 14 , wherein the IMC module further comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of a first input data in the input buffer; and
perform the operation on the first input data stored in the input buffer and second portions of a second operand stored in the array of DRAM rows.
17. The method of claim 14 , wherein the input data is divided into first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store one or more of the first portions and one or more of the second portions in a first row of the DRAM rows.
18. The method of claim 14 , wherein the IMC module comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of one or more of the first portions of the input data in the input buffer, the first portions of the input data being stored in a first row of the array of DRAM rows; and
perform the operation on the first portion stored in the input buffer and second tiles stored in the first row of the array of DRAM rows.
19. The method of claim 14 , wherein the input data comprises first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store a subset of the first portion in a first row of the array of DRAM rows and a subset of the second portions in a second row of the array of DRAM rows.
20. The method of claim 14 , wherein the IMC module further comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of the input data from a first row of the array of DRAM rows in the input buffer; and
perform the operation on the first portion stored in the input buffer and second portions of the input data stored in a second row of the array of DRAM rows.