Optimized hierarchical scratchpads for enhanced artificial intelligence accelerator core utilization

1. A method for optimized placement of data structures in a hierarchy of memory in a computing environment by a processor, comprising:

placing one or more data structures in a first scratchpad memory, a second scratchpad memory, an external memory, or a combination thereof in the hierarchy of memory according to a total memory capacity and bandwidth, a level of reuse of the one or more data structures, a number of operations that use each of the one or more data structures, a required duration each the one or more data structures are required to be placed a first scratchpad or a second scratchpad, and characteristics of those of the one or more data structures competing for placement in the hierarchy of memory that are able to co-exist at a same time step in at least one of a plurality of steps in training a deep neural network (DNN), wherein the second scratchpad memory is positioned between the external memory and the first scratchpad memory at one or more intermediary layers;

partitioning each of the external memory, the first scratchpad, and the second scratchpad into a first section to place static data structures and a second section to place temporal data structures;

placing static data structures, required for all timesteps of the plurality of training steps, into the first section; and

placing temporal data structures, required for only some timesteps of the plurality of training steps, into the second section.

2. The method of claim 1 , further including partitioning the second scratchpad into a plurality of accelerator core groups, wherein each accelerator core belonging to one of the plurality of accelerator core groups share a segment of the second scratchpad.

3. The method of claim 1 , further including:

assigning each accelerator core to the first scratchpad; or

relaxing the one or more data structures the first scratchpad memory, the second scratchpad memory, the external memory, or a combination thereof based on a capacity level.

4. The method of claim 1 , further including placing activation data structures into the first scratchpad, the second scratchpad, or the external memory according to a length of duration of the activation data structures.

5. The method of claim 1 , further including prioritizing placement of the one or more data structures into the first scratchpad, the second scratchpad, and the external memory according to a figure of merit (“FOM”), wherein the FOM is one or more of a plurality of factors that include at least a duration the one or more data structures should be retained as unused, a degree of memory boundness required for the one or more data structures, and a positive impact or negative impact upon each of the operations that use the one or more data structures.

6. The method of claim 5 , further including:

placing the one or more data structures into the first scratchpad according to the FOM of each of the one or more data structures;

placing the one or more data structures into the second scratchpad after the first scratchpad is full according to the FOM of each of the one or more data structures; or

placing the one or more data structures into the external memory after the first scratchpad, the second scratchpad, or a combination thereof are full according to the FOM of each of the one or more data structures.

7. A system for optimized placement of data structures in a hierarchy of memory in a computing environment, comprising:

one or more computers with executable instructions that when executed cause the system to:

place one or more data structures in a first scratchpad memory, a second scratchpad memory, an external memory, or a combination thereof in the hierarchy of memory according to a total memory capacity and bandwidth, a level of reuse of the one or more data structures, a number of operations that use each of the one or more data structures, a required duration each the one or more data structures are required to be placed a first scratchpad or a second scratchpad, and characteristics of those of the one or more data structures competing for placement in the hierarchy of memory that are able to co-exist at a same time step, wherein the second scratchpad memory is positioned between the external memory and the first scratchpad memory at one or more intermediary layers;

partition each of the external memory, the first scratchpad, and the second scratchpad into a first section to place static data structures and a second section to place temporal data structures;

place static data structures, required for all timesteps of the plurality of training steps, into the first section; and

place temporal data structures, required for only some timesteps of the plurality of training steps, into the second section.

8. The system of claim 7 , wherein the executable instructions partition the second scratchpad into a plurality of accelerator core groups, wherein each accelerator core belonging to one of the plurality of accelerator core groups share a segment of the second scratchpad.

9. The system of claim 7 , wherein the executable instructions:

assign each accelerator core to the first scratchpad; or

relax the one or more data structures the first scratchpad memory, the second scratchpad memory, the external memory, or a combination thereof based on a capacity level.

10. The system of claim 7 , wherein the executable instructions place activation data structures into the first scratchpad, the second scratchpad, or the external memory according to a length of duration of the activation data structures.

11. The system of claim 7 , wherein the executable instructions prioritize placement of the one or more data structures into the first scratchpad, the second scratchpad, and the external memory according to a figure of merit (“FOM”), wherein the FOM is one or more of a plurality of factors that include at least a duration the one or more data structures should be retained as unused, a degree of memory boundness required for the one or more data structures, and a positive impact or negative impact upon each of the operations that use the one or more data structures.

12. The system of claim 11 , wherein the executable instructions:

place the one or more data structures into the first scratchpad according to the FOM of each of the one or more data structures;

place the one or more data structures into the second scratchpad after the first scratchpad is full according to the FOM of each of the one or more data structures; or

place the one or more data structures into the external memory after the first scratchpad, the second scratchpad, or a combination thereof are full according to the FOM of each of the one or more data structures.

13. A computer program product for optimized placement of data structures in a hierarchy of memory by a processor in a computing environment, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

an executable portion that places one or more data structures in a first scratchpad memory, a second scratchpad memory, an external memory, or a combination thereof in the hierarchy of memory according to a total memory capacity and bandwidth, a level of reuse of the one or more data structures, a number of operations that use each of the one or more data structures, a required duration each the one or more data structures are required to be placed a first scratchpad or a second scratchpad, and characteristics of those of the one or more data structures competing for placement in the hierarchy of memory that are able to co-exist at a same time step, wherein the second scratchpad memory is positioned between the external memory and the first scratchpad memory at one or more intermediary layers;

an executable portion that partitions each of the external memory, the first scratchpad, and the second scratchpad into a first section to place static data structures and a second section to place temporal data structures;

an executable portion that places static data structures, required for all timesteps of the plurality of training steps, into the first section; and

an executable portion that places temporal data structures, required for only some timesteps of the plurality of training steps, into the second section.

14. The computer program product of claim 13 , further including an executable portion that partitions the second scratchpad into a plurality of accelerator core groups, wherein each accelerator core belonging to one of the plurality of accelerator core groups share a segment of the second scratchpad.

15. The computer program product of claim 13 , further including an executable portion that:

assigns each accelerator core to the first scratchpad; or

relaxes the one or more data structures the first scratchpad memory, the second scratchpad memory, the external memory, or a combination thereof based on a capacity level.

16. The computer program product of claim 13 , further including an executable portion that

places activation data structures into the first scratchpad, the second scratchpad, or the external memory according to a length of duration of the activation data structures.

17. The computer program product of claim 13 , further including an executable portion that:

prioritize placement of the one or more data structures into the first scratchpad, the second scratchpad, and the external memory according to a figure of merit (“FOM”), wherein the FOM is defined as a combination of a plurality of factors, wherein the FOM is one or more of a plurality of factors that include at least a duration the one or more data structures should be retained as unused, a degree of memory boundness required for the one or more data structures, and a positive impact or negative impact upon each of the operations that use the one or more data structures.

18. The computer program product of claim 17 , further including an executable portion that:

places the one or more data structures into the first scratchpad according to the FOM of each of the one or more data structures;

places the one or more data structures into the second scratchpad after the first scratchpad is full according to the FOM of each of the one or more data structures; or

places the one or more data structures into the external memory after the first scratchpad, the second scratchpad, or a combination thereof are full according to the FOM of each of the one or more data structures.