Systems and methods for exchange of data in distributed training of machine learning algorithms

1. A method of exchanging weights among a plurality of processors conducting training on a neural network (NN), the NN comprising neurons, the method comprising:

over a plurality of iterations:

at each of at least two processors of the plurality of processors:

sorting weights generated at the processors to create sorted weights;

compressing the sorted weights to produce compressed sorted weights; and

transmitting the compressed sorted weights to at least one master processor of the plurality of processors, the master processor updating a NN model based on the weights generated at the at least two processors;

wherein, on each Kth iteration of the plurality of iterations, K being an integer greater than 1:

a sort order of the sorted weights is created; and

the sort order is transmitted to least one processor of the plurality of processors.

2. The method of claim 1 , wherein on an iteration which is not a Kth iteration, the weights are sorted using a last sort order created.

3. The method of claim 1 , wherein K varies over time.

4. The method of claim 1 , wherein the neurons are artificial neurons represented by data stored by the plurality of processors.

5. The method of claim 1 , wherein the plurality of processors comprises a master processor and wherein the compressed sorted weights are transmitted to the master processor.

6. The method of claim 1 , wherein, the NN comprises a plurality of layers, and wherein during each iteration:

weights are generated for at least two of the plurality of layers, and

sorting, compressing, and transmitting is performed for the weights for the at least two of the plurality of layers.

7. The method of claim 1 , wherein, the NN comprises a plurality of layers and a plurality of links connecting the neurons, and wherein during each iteration the at least two processors execute a forward pass and a backward pass to generate gradients for the links and to generate weights for the links from the gradients.

8. A method of exchanging parameters among a plurality of computing nodes, the method comprising:

for a plurality of parameters, over a plurality of iterations including a periodically occurring ordering iteration:

at a first computing node:

if the iteration is an ordering iteration, arranging the parameters by value and storing an order of the arranging;

if the iteration is not an ordering iteration, arranging the parameters by a stored last order of the arranging;

compressing the arranged parameters to produce compressed parameters; and

sending the compressed parameters to a second computing node, the second computing node updating a NN model based on the parameters.

9. The method of claim 8 , wherein the ordering iteration occurs every Kth iteration, K being an integer.

10. The method of claim 9 , wherein K varies over time.

11. The method of claim 9 , wherein the parameters are weights for artificial neurons represented by data stored by at least one of the plurality of computing nodes.

12. The method of claim 9 , wherein the second computing node is a master node.

13. The method of claim 9 , wherein, the NN comprises a plurality of layers and a plurality of links connecting the neurons, and wherein during each iteration the node executes a forward pass and a backward pass to generate gradients for the links and to generate weights for the links from the gradients.

14. A system comprising a plurality of nodes conducting training on a neural network (NN), the NN comprising neurons, each node comprising:

a memory; and

a processor; the processor comprising at least one node configured to:

over a plurality of iterations:

sort weights generated at the processor to create sorted weights;

compress the sorted weights to produce compressed sorted weights; and

transmit the compressed sorted weights to least one other node of the plurality of nodes, the other node updating a NN model based on the weights generated at the processor;

wherein, on each Kth iteration of the plurality of iterations, K being an integer greater than 1:

a sort order of the sorted weights is created by the processor; and

the sort order is transmitted to least one node of the plurality of nodes.

15. The system of claim 14 , wherein on an iteration which is not a Kth iteration, the weights are sorted using a last sort order created.

16. The system of claim 15 , wherein K varies over time.

17. The system of claim 14 , wherein the neurons are artificial neurons represented by data stored by the plurality of nodes.

18. The system of claim 14 , wherein the plurality of nodes comprises a master node and wherein the compressed sorted weights are transmitted to the master node.

19. The system of claim 14 , wherein, the NN comprises a plurality of layers, and wherein during each iteration:

weights are generated for at least two of the plurality of layers, and

sorting, compressing, and transmitting is performed for the weights for the at least two of the plurality of layers.

20. The system of claim 14 , wherein, the NN comprises a plurality of layers and a plurality of links connecting the neurons, and wherein during each iteration the processor at least one node executes a forward pass and a backward pass to generate gradients for the links and to generate weights for the links from the gradients.