Gradient compression for distributed training
Techniques for exchanging compressed gradient data within a distributed system are disclosed. A set of gradients are computed at a first worker node of the distributed system using a neural network model and a set of weights associated with the neural network model. Each of the set of gradients having a value less than a threshold is clipped, resulting in non-clipped data elements and clipped data elements. A mapping indicating which of the set of gradients correspond to non-clipped data elements and which of the set of gradients correspond to clipped data elements is generated. Compressed data is generated based on the non-clipped data elements. The mapping and the compressed data are transmitted from the first worker node to a second worker node of the distributed system.
1. A method of exchanging compressed gradient data within a distributed system for training a neural network model, the method comprising:
computing, at a transmitting worker node of the distributed system, a set of gradients using the neural network model and a set of weights associated with the neural network model;
performing, at the transmitting worker node, a sparsity analysis on the set of gradients to determine a threshold;
clipping, by a direct memory access (DMA) controller at the transmitting worker node, at least 20% of the set of gradients by clipping each of the set of gradients having a value less than the threshold, resulting in the set of gradients comprising non-clipped data elements and clipped data elements, wherein a quantity of the set of gradients is greater than 1000;
generating, at the transmitting worker node, a mapping that indicates which of the set of gradients correspond to the non-clipped data elements and which of the set of gradients correspond to the clipped data elements, wherein the mapping includes a bitmap with binary values indicating locations of the non-clipped data elements and the clipped data elements;
generating, at the transmitting worker node, compressed data comprising the non-clipped data elements from the set of gradients;
transmitting the mapping including the bitmap and the compressed data from the transmitting worker node to a receiving worker node of the distributed system;
generating, at the receiving worker node, decompressed data by combining the non-clipped data elements from the compressed data with the clipped data elements using the mapping, such that the decompressed data includes the set of gradients comprising the non-clipped data elements and the clipped data elements; and
computing, at the receiving worker node, a set of synchronized gradients based on the set of gradients and other gradients received at the receiving worker node.
2. The method of claim 1 , further comprising:
forming, at the transmitting worker node, a header comprising the mapping and an original length of the set of gradients, the original length corresponding to a number of the non-clipped data elements and the clipped data elements.
3. The method of claim 1 , wherein clipping each of the set of gradients includes setting the value equal to zero such that the clipped data elements are zero data elements and the non-clipped data elements are non-zero data elements.
4. A method comprising:
computing, at a first worker node of a distributed system, a set of gradients using a neural network model and a set of weights associated with the neural network model;
clipping, by a direct memory access (DMA) controller at the first worker node, at least 20% of the set of gradients by clipping each of the set of gradients having a value less than a threshold, resulting in the set of gradients comprising non-clipped data elements and clipped data elements, wherein a quantity of the set of gradients is greater than 1000;
generating a mapping that indicates which of the set of gradients correspond to the non-clipped data elements and which of the set of gradients correspond to the clipped data elements, wherein the mapping includes a bitmap with binary values indicating locations of the non-clipped data elements and the clipped data elements;
generating compressed data based on the non-clipped data elements from the set of gradients; and
transmitting the mapping including the bitmap and the compressed data from the first worker node to a second worker node of the distributed system.
5. The method of claim 4 , further comprising:
generating, at the second worker node, decompressed data by combining the non-clipped data elements from the compressed data with the clipped data elements using the mapping to obtain the set of gradients comprising the non-clipped data elements and the clipped data elements.
6. The method of claim 4 , further comprising:
forming a header comprising the mapping, wherein the header and the compressed data are transmitted from the first worker node to the second worker node.
7. The method of claim 6 , wherein the header further comprises an original length of the set of gradients, the original length corresponding to a number of the non-clipped data elements and the clipped data elements.
8. The method of claim 4 , wherein clipping each of the set of gradients includes setting the value equal to zero such that the clipped data elements are zero data elements and the non-clipped data elements are non-zero data elements.
9. The method of claim 4 , further comprising:
performing a sparsity analysis on the set of gradients to determine the threshold.
10. The method of claim 9 , wherein performing the sparsity analysis includes:
calculating an average for the set of gradients;
calculating a standard deviation for the set of gradients; and
determining the threshold based on the average and the standard deviation.
11. A non-transitory computer-readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
computing, at a first worker node of a distributed system, a set of gradients using a neural network model and a set of weights associated with the neural network model;
clipping, by a direct memory access (DMA) controller at the first worker node, at least 20% of the set of gradients by clipping each of the set of gradients having a value less than a threshold, resulting in the set of gradients comprising non-clipped data elements and clipped data elements, wherein a quantity of the set of gradients is greater than 1000;
generating a mapping that indicates which of the set of gradients correspond to the non-clipped data elements and which of the set of gradients correspond to the clipped data elements, wherein the mapping includes a bitmap with binary values indicating locations of the non-clipped data elements and the clipped data elements;
generating compressed data based on the non-clipped data elements from the set of gradients; and
transmitting the mapping including the bitmap and the compressed data from the first worker node to a second worker node of the distributed system.
12. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise:
generating, at the second worker node, decompressed data by combining the non-clipped data elements from the compressed data with the clipped data elements using the mapping to obtain the set of gradients comprising the non-clipped data elements and the clipped data elements.
13. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise:
forming a header comprising the mapping, wherein the header and the compressed data are transmitted from the first worker node to the second worker node.
14. The non-transitory computer-readable medium of claim 13 , wherein the header further comprises an original length of the set of gradients, the original length corresponding to a number of the non-clipped data elements and the clipped data elements.
15. The non-transitory computer-readable medium of claim 11 , wherein clipping each of the set of gradients includes setting the value equal to zero such that the clipped data elements are zero data elements and the non-clipped data elements are non-zero data elements.
16. The non-transitory computer-readable medium of claim 11 , wherein the operations further comprise:
performing a sparsity analysis on the set of gradients to determine the threshold.
17. The non-transitory computer-readable medium of claim 16 , wherein performing the sparsity analysis includes:
calculating an average for the set of gradients;
calculating a standard deviation for the set of gradients; and
determining the threshold based on the average and the standard deviation.