Optimization of graphics processing unit memory for deep learning computing
Systems and methods are provided for optimizing GPU memory allocation for high-performance applications such as deep learning (DL) computing. For example, a DL task is executed using GPU resources (GPU device and GPU memory) to process a DL model having functional layers that are processed in a predefined sequence. A current functional layer of the DL model is invoked and processed using the GPU device. In response to the invoking, a data compression operation is performed to compress data of a previous functional layer of the DL model, and store the compressed data in the GPU memory. Responsive to the invoking, compressed data of a next functional layer of the DL model is accessed from the GPU memory and a data decompression operation is performed to decompress the compressed data for subsequent processing of the next functional layer of the DL model by the GPU device.
1. A method, comprising:
executing a deep learning task using graphics processing unit (GPU) resources on a computing node, wherein the GPU resources comprise at least one GPU device and GPU memory, and wherein the deep learning task is executed using a deep learning model comprising a plurality of functional layers that are processed in a predefined sequence;
wherein executing the deep learning task comprises:
invoking a current functional layer of the deep learning model, and processing, by the at least one GPU device, the current functional layer by using model data of the current functional layer to process output data generated by a previous functional layer; and
in response to the invoking of the current functional layer, performing a plurality of operations during processing of the current functional layer, wherein performing the plurality of operations during processing of the current functional layer comprises:
accessing the model data of the current functional layer for processing by the at least one GPU device, the accessed model data of the current functional layer comprising decompressed model data of the current functional layer generated by a data decompression process performed on compressed model data of the current functional layer during processing of the previous functional layer prior to the invoking of the current functional layer;
during processing of the current functional layer, invoking a compression process and a decompression process;
during processing of the current functional layer, performing a data compression operation by the at least one GPU device using the invoked compression process to compress model data of the previous functional layer of the deep learning model and store the compressed model data of the previous functional layer in the GPU memory; and
during processing of the current functional layer, accessing compressed model data of a next functional layer of the deep learning model from the GPU memory and performing a data decompression operation by the at least one GPU device using the invoked decompression process to decompress the compressed model data for subsequent processing of the next functional layer by the at least one GPU device in advance of invoking the next functional layer.
2. The method of claim 1 , wherein the deep learning task comprises a deep learning model training task comprising forward and backward propagation operations.
3. The method of claim 1 , wherein the processing of the current functional layer, the data compression operation, and the data decompression operation are executed as separate processing streams by the at least one GPU device.
4. The method of claim 1 , wherein the data compression operation and the data decompression operation are performed by loading and executing a compression kernel and a decompression kernel in the at least one GPU device.
5. The method of claim 1 , wherein the data compression operation and the data decompression operation are performed using hardware compression and decompression accelerators.
6. The method of claim 5 , wherein hardware compression and decompression accelerators are integrated within the at least one GPU device.
7. The method of claim 1 , further comprising maintaining metadata in the GPU memory, wherein the metadata comprises:
a layer structure of the deep learning model, wherein the layer structure comprises an identifier of each functional layer in the deep learning model and information regarding the predefined sequence of the functional layers of the deep learning model;
for each functional layer, data pointers to input data and output data of the functional layer stored in the GPU memory; and
for each functional layer, an indication of a current state of the input data and the output data of the functional layer, wherein the current state comprises one of a compressed state, a decompressed state, and a ready state.
8. The method of claim 7 , further comprising:
intercepting application programming interface (API) calls between the deep learning task executing on the computing node and the GPU resources; and
automatically determining the layer structure of the deep learning model using the intercepted API calls.
9. The method of claim 1 , wherein:
the functional layers of the deep learning model comprise a sequence of two or more convolutional layers;
the model data of the convolutional layers comprises respective filter parameters used for processing input data to the convolutional layers;
output data generated by the convolutional layers comprises feature maps; and
the filter parameters and feature maps of the convolutional layers are compressed and stored in the GPU memory.
10. An article of manufacture comprising a non-transitory processor-readable storage medium having stored program code of one or more software programs, wherein the program code is executable by one or more processors to implement method steps comprising:
executing a deep learning task using graphics processing unit (GPU) resources on a computing node, wherein the GPU resources comprise at least one GPU device and GPU memory, and wherein the deep learning task is executed using a deep learning model comprising a plurality of functional layers that are processed in a predefined sequence;
wherein executing the deep learning task comprises:
invoking a current functional layer of the deep learning model, and processing, by the at least one GPU device, the current functional layer by using model data of the current functional layer to process output data generated by a previous functional layer; and
in response to the invoking of the current functional layer, performing a plurality of operations during processing of the current functional layer, wherein performing the plurality of operations during processing of the current functional layer comprises:
accessing the model data of the current functional layer for processing by the at least one GPU device, the accessed model data of the current functional layer comprising decompressed model data of the current functional layer generated by a data decompression process performed on compressed model data of the current functional layer during processing of the previous functional layer prior to the invoking of the current functional layer;
during processing of the current functional layer, invoking a compression process and a decompression process;
during processing of the current functional layer, performing a data compression operation by the at least one GPU device using the invoked compression process to compress model data of the previous functional layer of the deep learning model and store the compressed model data of the previous functional layer in the GPU memory; and
during processing of the current functional layer, accessing compressed model data of a next functional layer of the deep learning model from the GPU memory and performing a data decompression operation by the at least one GPU device using the invoked decompression process to decompress the compressed model data for subsequent processing of the next functional layer by the at least one GPU device in advance of invoking the next functional layer.
11. The article of manufacture of claim 10 , wherein the processing of the current functional layer, the data compression operation, and the data decompression operation are executed as separate processing streams by the at least one GPU device.
12. The article of manufacture of claim 10 , wherein the data compression operation and the data decompression operation are performed by loading and executing a compression kernel and a decompression kernel in the at least one GPU device.
13. The article of manufacture of claim 10 , wherein the data compression operation and the data decompression operation are performed using hardware compression and decompression accelerators within the at least one GPU device.
14. The article of manufacture of claim 10 , further comprising executable program code for maintaining metadata in the GPU memory, wherein the metadata comprises:
a layer structure of the deep learning model, wherein the layer structure comprises an identifier of each functional layer in the deep learning model and information regarding the predefined sequence of the functional layers of the deep learning model;
for each functional layer, data pointers to input data and output data of the functional layer stored in the GPU memory; and
for each functional layer, an indication of a current state of the input data and the output data of the functional layer, wherein the current state comprises one of a compressed state, a decompressed state, and a ready state.
15. The article of manufacture of claim 14 , further comprising executable program code for:
intercepting application programming interface (API) calls between the deep learning task executing on the computing node and the GPU resources; and
automatically determining the layer structure of the deep learning model using the intercepted API calls.
16. The article of manufacture of claim 10 , wherein:
the functional layers of the deep learning model comprise a sequence of two or more convolutional layers;
the model data of the convolutional layers comprises respective filter parameters used for processing input data to the convolutional layers;
output data generated by the convolutional layers comprises feature maps; and
the filter parameters and feature maps of the convolutional layers are compressed and stored in the GPU memory.
17. A system, comprising:
a server node comprising;
graphics processing unit (GPU) resources comprising at least one GPU device and GPU memory,
system memory configured to store program code; and
at least one processor configured to execute the stored program code to perform a method comprising:
executing a deep learning task using the GPU resources, wherein the deep learning task is executed using a deep learning model comprising a plurality of functional layers that are processed in a predefined sequence;
wherein executing the deep learning task comprises:
invoking a current functional layer of the deep learning model, and processing, by the at least one GPU device, the current functional layer by using model data of the current functional layer to process output data generated by a previous functional layer; and
in response to the invoking of the current functional layer, performing a plurality of operations during processing of the current functional layer, wherein performing the plurality of operations during processing of the current functional layer comprises:
accessing the model data of the current functional layer for processing by the at least one GPU device, the accessed model data of the current functional layer comprising decompressed model data of the current functional layer generated by a data decompression process performed on compressed model data of the current functional layer during processing of the previous functional layer prior to the invoking of the current functional layer;
during processing of the current functional layer, invoking a compression process and a decompression process;
during processing of the current functional layer, performing a data compression operation by the at least one GPU device using the invoked compression process to compress model data of the previous functional layer of the deep learning model and store the compressed model data of the previous functional layer in the GPU memory; and
during processing of the current functional layer, accessing compressed model data of a next functional layer of the deep learning model from the GPU memory and performing a data decompression operation by the at least one GPU device using the invoked decompression process to decompress the compressed model data for subsequent processing of the next functional layer by the at least one GPU device in advance of invoking the next functional layer.
18. The system of claim 17 , wherein the processing of the current functional layer, the data compression operation, and the data decompression operation are executed as separate processing streams by the at least one GPU device, and wherein the data compression operation and the data decompression operation are performed by loading and executing a compression kernel and a decompression kernel in the at least one GPU device.
19. The system of claim 17 , wherein the at least one processor is configured to execute the stored program code to instantiate a memory manager which is configured to maintain metadata in the GPU memory, wherein the metadata comprises:
a layer structure of the deep learning model, wherein the layer structure comprises an identifier of each functional layer in the deep learning model and information regarding the predefined sequence of the functional layers of the deep learning model;
for each functional layer, data pointers to input data and output data of the functional layer stored in the GPU memory; and
for each functional layer, an indication of a current state of the input data and the output data of the functional layer, wherein the current state comprises one of a compressed state, a decompressed state, and a ready state.
20. The system of claim 17 , wherein the system comprises a cloud computing system which is configured to implement at least one service, wherein the at least one service comprises at least one of GPU-as-a-Service and Deep Learning-as-a-Service.