Choosing execution mode of a neural network based on total memory usage
A computer-implemented method, a computer program product, and a computer processing system are provided for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold size. The multiple GPU execution modes include a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode. The method includes starting an execution on the NN with the UM mode and measuring the memory usage for each of layers of the NN. The method further includes selecting an execution mode based on the memory usage of all of the layers.
1. A computer-implemented method for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold physical GPU memory size, the multiple GPU execution modes comprising a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode, the method comprising:
in a multiple execution mode selection scheme having any of the multiple GPU execution modes of the normal memory mode, the OoC execution mode, and the UM mode available for initial execution of all layers of the NN, performing an initial iteration by executing all of the layers of the NN with the UM mode from among the multiple GPU execution modes to ensure memory coverage of the NN;
for each iteration of executing all of the layers of the NN:
measuring memory usage for each layer of the NN;
changing to a UM mode by copying data from a non-unified memory to a UM responsive to an occurrence of a memory shortage when executing in the normal memory mode or OoC execution mode;
determining a total memory usage by all of the layers of the NN and a maximum layer memory usage; and
selecting from the multiple GPU execution modes a GPU execution mode for each of the layers of the NN to execute with in a next iteration, wherein the normal memory mode is selected responsive to the total memory usage by all of the layers of the NN being less than a first predetermined threshold and the OoC execution mode is selected responsive to the total memory usage by all of the layers of the NN being greater than or equal to the first predetermined threshold and the maximum layer memory usage being less than a second predetermined threshold.
2. The computer-implemented method of claim 1 , wherein different ones of the multiple GPU execution modes can be selected for different ones of the layers.
3. The computer-implemented method of claim 1 , wherein different ones of the multiple GPU execution modes can be selected for forward phases and backward phases.
4. A non-transitory computer program product for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold memory size, the multiple GPU execution modes comprising a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
in a multiple execution mode selection scheme having any of the multiple GPU execution modes of the normal memory mode, the OoC execution mode, and the UM mode available for initial execution of all layers of the NN, performing an initial iteration by executing all of the layers of the NN with the UM mode from among the multiple GPU execution modes to ensure memory coverage of the NN;
for each iteration of executing all of the layers of the NN:
measuring memory usage for each layer of the NN;
changing to a UM mode by copying data from a non-unified memory to a UM responsive to an occurrence of a memory shortage when executing in the normal memory mode or OoC execution mode;
determining a total memory usage by all of the layers of the NN and a maximum layer memory usage; and
selecting from the multiple GPU execution modes a GPU execution mode for each of the layers of the NN to execute with in a next iteration, wherein the normal memory mode is selected responsive to the total memory usage by all of the layers of the NN being less than a first predetermined threshold and the OoC execution mode is selected responsive to the total memory usage by all of the layers of the NN being greater than or equal to the first predetermined threshold and the maximum layer memory usage being less than a second predetermined threshold.
5. The non-transitory computer program product of claim 4 , wherein different ones of the multiple GPU execution modes can be selected for different ones of the layers.
6. The non-transitory computer program product of claim 4 , wherein different ones of the multiple GPU execution modes can be selected for forward phases and backward phases.
7. A computer processing system for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold physical GPU memory size, the multiple GPU execution modes comprising a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode, the system comprising:
at least one GPU;
a memory for storing program code;
a processing element for running the program code to
in a multiple execution mode selection scheme having any of the multiple GPU execution modes of the normal memory mode, the OoC execution mode, and the UM mode available for initial execution of all layers of the NN, perform an initial iteration by executing all of the layers of the NN with the UM mode from among the multiple GPU execution modes;
for each iteration of executing all of the layers of the NN:
measure memory usage for each layer of the NN;
change to a UM mode by copying data from a non-unified memory to a UM responsive to an occurrence of a memory shortage when executing in the normal memory mode or OoC execution mode;
determine a total memory usage by all of the layers of the NN and a maximum layer usage; and
select from the multiple GPU execution modes a GPU execution mode for each of the layers of the NN to execute with in a next iteration, wherein the normal memory mode is selected responsive to the total memory usage by all of the layers of the NN being less than a first predetermined threshold and the OoC execution mode is selected responsive to the total memory usage by all of the layers of the NN being greater than or equal to the first predetermined threshold and the maximum layer memory usage being less than a second predetermined threshold.
8. The computer processing system of claim 7 , wherein different ones of the multiple GPU execution modes can be selected for different ones of the layers.