Gradient pruning for efficient training of machine learning models
Certain aspects of the present disclosure provide techniques for improved machine learning using gradient pruning, comprising computing, using a first batch of training data, a first gradient tensor comprising a gradient for each parameter of a parameter tensor for a machine learning model; identifying a first subset of gradients in the first gradient tensor based on a first gradient criteria; and updating a first subset of parameters in the parameter tensor based on the first subset of gradients in the first gradient tensor.
1. A method for training a machine learning model, comprising:
computing using a first batch of training data, a first gradient tensor comprising a gradient for each parameter of a parameter tensor for a machine learning model;
identifying a first subset of gradients in the first gradient tensor based on evaluating each respective gradient of the first gradient tensor using a first gradient criteria; and
updating a first subset of parameters in the parameter tensor based on the first subset of gradients.
2. The method of claim 1 , further comprising: generating a sparse gradient tensor based on the first subset of gradients in the first gradient tensor.
3. The method of claim 2 , further comprising: transmitting the sparse gradient tensor to a remote server managing federated learning of the machine learning model.
4. The method of claim 2 , wherein the sparse gradient tensor is stored in a compressed sparse row (CSR) format.
5. The method of claim 1 , further comprising:
computing, using a second batch of training data, a second gradient tensor comprising a gradient for each parameter of the parameter tensor;
determining a first set of indices corresponding to the first subset of gradients in the first gradient tensor; and
updating the first subset of parameters in the parameter tensor based on the first set of indices and the second gradient tensor.
6. The method of claim 1 , further comprising:
computing, using a second batch of training data, a second gradient tensor comprising a gradient for each parameter of the parameter tensor; and
identifying a second subset of gradients in the second gradient tensor based on a second gradient criteria.
7. The method of claim 6 , wherein the first subset of gradients comprises more gradient values than the second subset of gradients.
8. The method of claim 6 , wherein the first subset of gradients comprises fewer gradient values than the second subset of gradients.
9. The method of claim 1 , wherein the first gradient criteria comprises a percentile value.
10. The method of claim 1 , wherein updating the first subset of parameters in the parameter tensor based on the first subset of gradients is performed during a training iteration and comprises modifying only parameters in the parameter tensor that correspond to gradients in the first subset of gradients, wherein at least one parameter in the parameter tensor that does not correspond to any gradient in the first subset of gradients is not updated during the training iteration.
11. The method of claim 1 , further comprising deploying the machine learning model to generate model outputs.
12. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform an operation comprising:
computing using a first batch of training data, a first gradient tensor comprising a gradient for each parameter of a parameter tensor for a machine learning model;
identifying a first subset of gradients in the first gradient tensor based on evaluating each respective gradient of the first gradient tensor using a first gradient criteria; and
updating a first subset of parameters in the parameter tensor based on the first subset of gradients.
13. The non-transitory computer-readable medium of claim 12 , wherein the operation further comprises: generating a sparse gradient tensor based on the first subset of gradients in the first gradient tensor.
14. The non-transitory computer-readable medium of claim 13 , wherein the operation further comprises: transmitting the sparse gradient tensor to a remote server managing federated learning of the machine learning model.
15. The non-transitory computer-readable medium of claim 12 , wherein the operation further comprises:
computing, using a second batch of training data, a second gradient tensor comprising a gradient for each parameter of the parameter tensor;
determining a first set of indices corresponding to the first subset of gradients in the first gradient tensor; and
updating the first subset of parameters in the parameter tensor based on the first set of indices and the second gradient tensor.
16. The computer-readable medium of claim 12 , the operation further comprising:
computing, using a second batch of training data, a second gradient tensor comprising a gradient for each parameter of the parameter tensor; and
identifying a second subset of gradients in the second gradient tensor based on a second gradient criteria.
17. The computer-readable medium of claim 16 , wherein the first subset of gradients comprises more gradient values than the second subset of gradients.
18. The computer-readable medium of claim 16 , wherein the first subset of gradients comprises fewer gradient values than the second subset of gradients.
19. The computer-readable medium of claim 12 , wherein the first gradient criteria comprises a percentile value.
20. The computer-readable medium of claim 12 , wherein updating the first subset of parameters in the parameter tensor based on the first subset of gradients is performed during a training iteration and comprises modifying only parameters in the parameter tensor that correspond to gradients in the first subset of gradients, wherein at least one parameter in the parameter tensor that does not correspond to any gradient in the first subset of gradients is not updated during the training iteration.
21. A processing system, comprising:
a memory comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the processing system to perform an operation comprising:
computing, using a first batch of training data, a first gradient tensor comprising a gradient for each parameter of a parameter tensor for a machine learning model;
identifying a first subset of gradients in the first gradient tensor based on evaluating each respective gradient of the first gradient tensor using a first gradient criteria; and
updating a first subset of parameters in the parameter tensor based on the first subset of gradients.
22. The processing system of claim 21 , wherein the operation further comprises: generating a sparse gradient tensor based on the first subset of gradients in the first gradient tensor.
23. The processing system of claim 22 , wherein the operation further comprises: transmitting the sparse gradient tensor to a remote server managing federated learning of the machine learning model.
24. The processing system of claim 21 , wherein the operation further comprises:
computing, using a second batch of training data, a second gradient tensor comprising a gradient for each parameter of the parameter tensor;
determining a first set of indices corresponding to the first subset of gradients in the first gradient tensor; and
updating the first subset of parameters in the parameter tensor based on the first set of indices and the second gradient tensor.
25. The processing system of claim 21 , wherein the operation further comprises:
computing, using a second batch of training data, a second gradient tensor comprising a gradient for each parameter of the parameter tensor; and
identifying a second subset of gradients in the second gradient tensor based on a second gradient criteria.
26. The processing system of claim 25 , wherein the first subset of gradients comprises more gradient values than the second subset of gradients.
27. The processing system of claim 25 , wherein the first subset of gradients comprises fewer gradient values than the second subset of gradients.
28. The processing system of claim 21 , wherein the first gradient criteria comprises a percentile value.
29. The processing system of claim 21 , wherein updating the first subset of parameters in the parameter tensor based on the first subset of gradients is performed during a training iteration and comprises modifying only parameters in the parameter tensor that correspond to gradients in the first subset of gradients, wherein at least one parameter in the parameter tensor that does not correspond to any gradient in the first subset of gradients is not updated during the training iteration.
30. A processing system configured to train a machine learning model, comprising:
means for computing, using a first batch of training data, a first gradient tensor comprising a gradient for each parameter of a parameter tensor for a machine learning model;
means for identifying a first subset of gradients in the first gradient tensor based on evaluating each respective gradient of the first gradient tensor using a first gradient criteria; and
means for updating a first subset of parameters in the parameter tensor based on the first subset of gradients.