Compressing weights for distributed neural networks

1. A system comprising:

a set of processing units; and

a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:

train a first network comprising a first set of weights using a set of training data;

train a second network comprising a second set of weights using the set of training data, wherein a number of weights in the first set of weights is greater than a number of weights in the second set of weights;

adjust the first set of weights based on a first loss determined by the first network and a second loss determined by the second network;

adjust the second set of weights based on the first loss determined by the first network and the second loss determined by the second network; and

send values of the adjusted second set of weights to a computing system.

2. The system of claim 1 , wherein the instructions further cause the at least one processing unit to:

receive a set of values from the computing system; and

use the set of values as the values of the second set of weights.

3. The system of claim 2 , wherein the set of values is a set of average values calculated from the second set of weights of the second network and values of weights that the computing system received from other systems.

4. The system of claim 2 , wherein the set of training data is a first set of training data, wherein the instructions further cause the at least one processing unit to:

train the first network using the second set of training data;

train the second network using the second set of training data;

adjust the first set of weights based on a third loss determined by the first network and a fourth loss determined by the second network; and

adjust the second set of weights based on the third loss determined by the first network and the fourth loss determined by the second network.

5. The system of claim 1 , wherein adjusting the first set of weights comprises:

subtracting the second loss from the first loss to form a difference;

adding the difference to the first loss to form a sum; and

adjusting the first set of weights based on the sum.

6. The system of claim 1 , wherein adjusting the second set of weights comprises:

subtracting the first loss from the second loss to form a difference; and

adjusting the second set of weights based on the difference.

7. The system of claim 1 , wherein the instructions further cause the at least one processing unit to:

generate a first output based on training the first network using the set of training data; and

generate a second output based on training the second network using the set of training data,

wherein the first network determines the first loss based on the set of training data and the first output, wherein the second network determines the second loss based on the set of training data and the second output.

8. A method comprising:

training a first network comprising a first set of weights using a set of training data;

training a second network comprising a second set of weights using the set of training data, wherein a number of weights in the first set of weights is greater than a number of weights in the second set of weights;

adjusting the first set of weights based on a first loss determined by the first network and a second loss determined by the second network;

adjusting the second set of weights based on the first loss determined by the first network and the second loss determined by the second network; and

sending values of the adjusted second set of weights to a computing system.

9. The method of claim 8 further comprising:

receiving a set of values from the computing system; and

using the set of values as the values of the second set of weights.

10. The method of claim 9 , wherein the set of values is a set of average values calculated from the second set of weights of the second network and values of weights that the computing system received from other systems.

11. The method of claim 9 , wherein the set of training data is a first set of training data, the method further comprising:

training the first network using the second set of training data;

training the second network using the second set of training data;

adjusting the first set of weights based on a third loss determined by the first network and a fourth loss determined by the second network; and

adjusting the second set of weights based on the third loss determined by the first network and the fourth loss determined by the second network.

12. The method of claim 8 , wherein adjusting the first set of weights comprises:

subtracting the second loss from the first loss to form a difference;

adding the difference to the first loss to form a sum; and

adjusting the first set of weights based on the sum.

13. The method of claim 8 , wherein adjusting the second set of weights comprises:

subtracting the first loss from the second loss to form a difference; and

adjusting the second set of weights based on the difference.

14. The method of claim 8 further comprising:

generating a first output based on training the first network using the set of training data; and

generating a second output based on training the second network using the set of training data,

wherein the first network determines the first loss based on the set of training data and the first output, wherein the second network determines the second loss based on the set of training data and the second output.

15. A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:

training a first network comprising a first set of weights using a set of training data;

training a second network comprising a second set of weights using the set of training data, wherein a number of weights in the first set of weights is greater than a number of weights in the second set of weights;

adjusting the first set of weights based on a first loss determined by the first network and a second loss determined by the second network;

adjusting the second set of weights based on the first loss determined by the first network and the second loss determined by the second network; and

sending values of the adjusted second set of weights to a computing system.

16. The non-transitory machine-readable medium of claim 15 , wherein the program further comprises sets of instructions for:

receiving a set of values from the computing system; and

using the set of values as the values of the second set of weights.

17. The non-transitory machine-readable medium of claim 16 , wherein the set of values is a set of average values calculated from the second set of weights of the second network and values of weights that the computing system received from other systems.

18. The non-transitory machine-readable medium of claim 16 , wherein the set of training data is a first set of training data, wherein the program further comprises sets of instructions for:

training the first network using the second set of training data;

training the second network using the second set of training data;

adjusting the first set of weights based on a third loss determined by the first network and a fourth loss determined by the second network; and

adjusting the second set of weights based on the third loss determined by the first network and the fourth loss determined by the second network.

19. The non-transitory machine-readable medium of claim 15 , wherein adjusting the first set of weights comprises:

subtracting the second loss from the first loss to form a difference;

adding the difference to the first loss to form a sum; and

adjusting the first set of weights based on the sum.

20. The non-transitory machine-readable medium of claim 15 , wherein adjusting the second set of weights comprises:

subtracting the first loss from the second loss to form a difference; and

adjusting the second set of weights based on the difference.