ReLU compression to reduce GPU memory
A method is presented for compressing data of a Rectified Linear Unit (ReLU) function on a graphical processing unit (GPU) employed in a learning process of a deep neural network. The method includes converting an initial data structure including nonzero data and zero data into a compressed data structure including only the nonzero data of the initial data structure as compressed data by generating a nonzero data bitmap region, generating a nonzero data number table region by employing a parallel reduction algorithm, calculating a nonzero data array index per block region of all blocks from the nonzero data number table region by employing a parallel prefix sum scan algorithm, allocating a buffer for the compressed data; and copying the nonzero data from the initial data structure into a nonzero data array region in a compressed data format in parallel.
1. A method for compressing data of a Rectified Linear Unit (ReLU) function on a graphical processing unit (GPU) employed in a learning process of a deep neural network, the method comprising:
converting an initial data structure including nonzero data and zero data into a compressed data structure including only the nonzero data of the initial data structure as compressed data by:
generating a nonzero data bitmap region;
generating a nonzero data number table region by employing a parallel reduction algorithm;
calculating a nonzero data array index per block region of all blocks from the nonzero data number table region by employing a parallel prefix sum scan algorithm;
allocating a buffer for the compressed data; and
copying the nonzero data from the initial data structure into a nonzero data array region in a compressed data format in parallel.
2. The method of claim 1 , wherein the compressed data is decompressed by allocating the buffer from the nonzero data and zero data of the initial data structure, and then clearing the buffer.
3. The method of claim 2 , wherein the decompression further includes calculating the nonzero data array index per block region of all blocks from the nonzero data number table by employing the parallel prefix sum scan algorithm.
4. The method of claim 3 , wherein the decompression further includes copying the nonzero data array from the initial data structure into the nonzero data bitmap region in parallel.
5. The method of claim 1 , wherein the nonzero data bitmap region is displayed directly above the nonzero data array index per block region in the compressed data structure.
6. The method of claim 1 , wherein the nonzero data number table region is displayed directly above the nonzero data bitmap region in the compressed data structure.
7. The method of claim 1 , wherein the nonzero data bitmap region displays the uncompressed data in binary format in a sequential manner.
8. A non-transitory computer-readable storage medium comprising a computer-readable program for compressing data of a Rectified Linear Unit (ReLU) function on a graphical processing unit (GPU) employed in a learning process of a deep neural network, wherein the computer-readable program when executed on a computer causes the computer to:
convert an initial data structure including nonzero data and zero data into a compressed data structure including only the nonzero data of the initial data structure as compressed data by:
generate a nonzero data bitmap region;
generate a nonzero data number table region by employing a parallel reduction algorithm;
calculate a nonzero data array index per block region of all blocks from the nonzero data number table region by employing a parallel prefix sum scan algorithm;
allocate a buffer for the compressed data; and
copy the nonzero data from the initial data structure into a nonzero data array region in a compressed data format in parallel.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the compressed data is decompressed by allocating the buffer from the nonzero data and zero data of the initial data structure, and then clearing the buffer.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the decompression further includes calculating the nonzero data array index per block region of all blocks from the nonzero data number table by employing the parallel prefix sum scan algorithm.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the decompression further includes copying the nonzero data array from the initial data structure into the nonzero data bitmap region in parallel.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the nonzero data bitmap region is displayed directly above the nonzero data array index per block region in the compressed data structure.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the nonzero data number table region is displayed directly above the nonzero data bitmap region in the compressed data structure.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the nonzero data bitmap region displays the uncompressed data in binary format in a sequential manner.
15. A system for compressing data of a Rectified Linear Unit (ReLU) function on a graphical processing unit (GPU) employed in a learning process of a deep neural network, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
convert an initial data structure including nonzero data and zero data into a compressed data structure including only the nonzero data of the initial data structure as compressed data by:
generating a nonzero data bitmap region;
generating a nonzero data number table region by employing a parallel reduction algorithm;
calculating a nonzero data array index per block region of all blocks from the nonzero data number table region by employing a parallel prefix sum scan algorithm;
allocating a buffer for the compressed data; and
copying the nonzero data from the initial data structure into a nonzero data array region in a compressed data format in parallel.
16. The system of claim 15 , wherein the compressed data is decompressed by allocating the buffer from the nonzero data and zero data of the initial data structure, and then clearing the buffer.
17. The system of claim 16 , wherein the decompression further includes calculating the nonzero data array index per block region of all blocks from the nonzero data number table by employing the parallel prefix sum scan algorithm.
18. The system of claim 17 , wherein the decompression further includes copying the nonzero data array from the initial data structure into the nonzero data bitmap region in parallel.
19. The system of claim 15 , wherein the nonzero data bitmap region is displayed directly above the nonzero data array index per block region in the compressed data structure.
20. The system of claim 15 ,
wherein the nonzero data number table region is displayed directly above the nonzero data bitmap region in the compressed data structure; and
wherein the nonzero data bitmap region displays the uncompressed data in binary format in a sequential manner.