NVM-based high-capacity neural network inference engine
A system, method, and computer program product for a neural network inference engine is disclosed. The inference engine system may include a first memory and a processor in communication with the first memory. The processor may be configured to perform operations. The operations the processor is configured to perform may include fetching a first task with said first memory and delivering the first task to the processor for processing the first task. The operations may further include prefetching a second task with the first memory while the processor is processing the first task. The operations may further include the first memory delivering the second task to the processor upon completion of processing the first task. The operations may further include the processor processing the second task.
1 . A computer system comprising:
a memory stack comprising a buffer die and a plurality of memory dies, including a first memory die, wherein:
the buffer die and each memory die of the plurality of memory dies are connected via vertical interconnect; and
the buffer die comprises artificial intelligence cores, a first buffer segment, and a second buffer segment;
a processor set;
one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media to cause the processor to perform operations comprising:
fetching a first task from the first memory die;
delivering, by the first buffer segment, the first task to said processor;
prefetching, with the second buffer segment, a second task from a memory die of the plurality of memory dies while the processor is processing the first task;
delivering, by the second buffer segment and upon completion of processing the first task, the second task to the processor; and
processing, by the processor, the second task.
2 . The computer system of claim 1 , wherein the operations further comprise:
responsive to the processor processing the first task, generating a first task computation;
sending, by the processor, the first task computation to the second buffer segment;
accepting, by the second buffer segment, first task computation; and
delivering the first task computation to the first memory die, wherein the first memory die is a higher-density memory than the second buffer segment.
3 . The computer system of claim 2 , wherein:
the first memory die is integrated in a three-dimensional stack of memory, wherein the three-dimensional stack of memory comprises the plurality of memory dies.
4 . The computer system of claim 1 , wherein the operations further comprise:
sensing, with a first temperature sensor located on the first memory die, a first temperature, wherein:
the first temperature sensor is in communication with a power gate; and
the power gate throttles power to the first memory responsive to a first temperature threshold being reached; and
sensing, with a second temperature sensor located on the buffer die, a second temperature.
5 . The computer system of claim 4 , wherein
the second temperature sensor is in communication with the power gate; and
the power gate throttles power to the first memory responsive to a second temperature threshold being reached.
6 . The computer system of claim 1 , wherein the operations further comprise:
communicating between an error correction engine, the first memory die, and the processor, wherein data bits and check bits for the error correction engine are co-located.
7 . A computer implemented method comprising:
providing a memory stack comprising a buffer die and a plurality of memory dies, including a first memory die, wherein:
the buffer die and each memory die of the plurality of memory dies are connected via vertical interconnect; and
the buffer die comprises artificial intelligence cores, a first buffer segment, and a second buffer segment;
fetching a first task from the first memory die;
delivering, by the first buffer segment, the first task to a processor;
prefetching, with the second buffer segment, a second task from a memory die of the plurality of memory dies while the processor is processing the first task;
delivering, by the second buffer segment and upon completion of processing the first task, the second task to the processor; and
processing, by the processor, the second task.
8 . The computer implemented method of claim 7 , further comprising:
responsive to the processor processing the first task, generating a first task computation;
sending, by the processor, the first task computation to the second buffer segment;
accepting, by the second buffer segment, the first task computation; and
delivering the first task computation to the first memory die, wherein the first memory die is a higher-density memory than the second buffer segment.
9 . The computer implemented method of claim 8 wherein:
the first memory die is integrated in a three-dimensional stack of memory, wherein the three-dimensional stack of memory comprises the plurality of memory dies.
10 . The computer implemented method of claim 7 further comprising:
sensing, with a first temperature sensor located on the first memory die, a first temperature, wherein:
the first temperature sensor is in communication with a power gate; and
the power gate throttles power to the first memory responsive to a first temperature threshold being reached; and
sensing, with a second temperature sensor located on the buffer die, a second temperature.
11 . The computer implemented method of claim 10 , wherein;
the second temperature sensor is in communication with the power gate; and
the power gate throttles power to the first memory responsive to a second temperature threshold being reached.
12 . The computer implemented method of claim 7 , further comprising:
communicating between an error correction engine, the first memory die, and the processor, wherein data bits and check bits for the error correction engine are co-located.
13 . A computer program product comprising:
one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media to perform operations comprising:
providing a memory stack comprising a buffer die and a plurality of memory dies, including a first memory die, wherein:
the buffer die and each memory die of the plurality of memory dies are connected via vertical interconnect; and
the buffer die comprises artificial intelligence cores, a first buffer segment, and a second buffer segment;
fetching a first task from the first memory die;
delivering, by the first buffer segment, the first task to a processor;
prefetching, with the second buffer segment, a second task from a memory die of the plurality of memory dies while the processor is processing the first task;
delivering, by the second buffer segment and upon completion of processing the first task, the second task to the processor; and
processing, by the processor, the second task.
14 . The computer program product of claim 13 , wherein the operations further comprise:
responsive to the processor processing the first task, generating a first task computation;
sending, by the processor, the first task computation to the second buffer segment;
accepting, by the second buffer segment, the first task computation; and
delivering the first task computation to the first memory die, wherein the first memory die is a higher-density memory than the second buffer segment.
15 . The computer program product of claim 13 , wherein the operations further comprise:
sensing, with a first temperature sensor located on the first memory die, a first temperature, wherein:
the first temperature sensor is in communication with a power gate; and
the power gate throttles power to the first memory
responsive to a first temperature threshold being reached; and
sensing, with a second temperature sensor located on the buffer die, a second temperature.
16 . The computer system of claim 1 , wherein the processor is in the buffer die.
17 . The computer system of claim 1 , wherein the operations further comprise:
holding the second task in the second buffer segment while the processor is processing the first task.
18 . The computer system of claim 1 , wherein the processor receives the first task from the first buffer segment.