Coordinated learning using distributed average consensus

1. A method for cooperative learning comprising:

generating, at a distributed computing device, a gradient descent matrix based on data received by the distributed computing device and a model stored on the distributed computing device;

calculating, by the distributed computing device, a sampled gradient descent matrix based on the gradient descent matrix and a random matrix;

iteratively executing, by the distributed computing device, a process to determine a consensus gradient descent matrix in conjunction with a plurality of additional distributed computing devices connected to the distributed computing device by a network, the consensus gradient descent matrix based on the sampled gradient descent matrix calculated by the distributed computing device and a plurality of additional sampled gradient descent matrices calculated by the plurality of additional distributed computing devices; and

updating, by the distributed computing device, the model stored on the distributed computing device based on the consensus gradient descent matrix.

2. The method of claim 1 , wherein iteratively executing the process to determine the consensus gradient descent matrix comprises, for a first iteration of the process:

transmitting, over the network, the sampled gradient descent matrix of the distributed computing device to a second distributed computing device of the plurality of additional distributed computing devices;

receiving, over the network, an additional sampled gradient descent matrix generated by the second distributed computing device from the second distributed computing device; and

calculating the consensus gradient descent matrix by computing an average of the sampled gradient descent matrix and the additional sampled gradient descent matrix.

3. The method of claim 2 , wherein iteratively executing the process to determine the consensus gradient descent matrix comprises, for a second iteration of the process:

transmitting, over the network, the consensus gradient descent matrix of the distributed computing device to a third distributed computing device of the plurality of additional distributed computing devices;

receiving, over the network, an additional consensus gradient descent matrix generated by the third distributed computing device from the third distributed computing device; and

updating the consensus gradient descent matrix by computing an average of the consensus gradient descent matrix and the additional consensus gradient descent matrix.

4. The method of claim 1 , wherein iteratively executing the process to determine a consensus gradient descent matrix comprises, for each iteration of the process:

transmitting, over the network, a first convergence indicator of the distributed computing device to a second distributed computing device of the plurality of additional distributed computing devices;

receiving, over the network, a second convergence indicator of the second distributed computing device from the second distributed computing device;

updating the first convergence indicator of the distributed computing device by determining a center of mass of the first convergence indicator and the second convergence indicator; and

determining whether the consensus gradient descent matrix has been obtained based on the updated first convergence indicator.

5. The method of claim 4 , wherein determining whether the consensus gradient descent matrix has been obtained based on the updated first convergence indicator comprises determining that the first convergence indicator of the distributed computing device is within a threshold distance of a global center of mass of the first convergence indicator.

6. The method of claim 1 , wherein generating the gradient descent matrix based on data received by the distributed computing device and a model stored on the distributed computing device comprises:

receiving, at the distributed computing device, a plurality of pairs of training data, each pair comprising a data input and a label;

for each pair of the plurality of pairs of training data, computing a gradient vector of a plurality of gradient vectors by evaluating a partial derivative of an objective function of the model based on the pair; and

concatenating the plurality of gradient vectors to generate the gradient descent matrix.

7. The method of claim 1 , wherein the sampled gradient descent matrix represents the gradient descent matrix in a cooperative subspace common to the distributed computing devices and the plurality of additional distributed computing devices.

8. The method of claim 1 , wherein calculating the sampled gradient descent matrix based on the gradient descent matrix and a random matrix comprises:

generating, as the random matrix, a Gaussian ensemble matrix; and

multiplying the gradient descent matrix and the Gaussian ensemble matrix to generate the sampled gradient descent matrix.

9. The method of claim 1 , wherein updating the model stored on the distributed computing device based on the consensus gradient descent matrix comprises:

extracting an orthogonal subspace of the consensus gradient descent matrix spanning the range of a global gradient descent matrix; and

updating weights of the model based on the extracted orthogonal subspace.

10. The method of claim 1 , wherein the model is a machine learning artificial intelligence (AI) model configured to make a prediction based on one or more input signals received by the distributed computing device.

11. A non-transitory computer readable storage medium configured to store program code, the program code comprising instructions that, when executed by one or more processors, cause the one or more processors to:

generate a gradient descent matrix based on data received by a distributed computing device and a model stored on the distributed computing device;

calculate a sampled gradient descent matrix based on the gradient descent matrix and a random matrix;

iteratively execute a process to determine a consensus gradient descent matrix in conjunction with a plurality of additional distributed computing devices connected by a network to the distributed computing device, the consensus gradient descent matrix based on the sampled gradient descent matrix calculated by the distributed computing device and a plurality of additional sampled gradient descent matrices calculated by the plurality of additional distributed computing devices; and

update the model stored on the distributed computing device based on the consensus gradient descent matrix.

12. The non-transitory computer readable storage medium of claim 11 , wherein the instructions to iteratively execute the process to determine the consensus gradient descent matrix comprise instructions to, for a first iteration of the process:

transmit, over the network, the sampled gradient descent matrix of the distributed computing device to a second distributed computing device of the plurality of additional distributed computing devices;

receive, over the network, an additional sampled gradient descent matrix generated by the second distributed computing device from the second distributed computing device; and

calculate the consensus gradient descent matrix by computing an average of the sampled gradient descent matrix and the additional sampled gradient descent matrix.

13. The non-transitory computer readable storage medium of claim 12 , wherein the instructions to iteratively execute the process to determine the consensus gradient descent matrix further comprise instructions to, for a second iteration of the process:

transmit, over the network, the consensus gradient descent matrix of the distributed computing device to a third distributed computing device of the plurality of additional distributed computing devices;

receive, over the network, an additional consensus gradient descent matrix generated by the third distributed computing device from the third distributed computing device; and

update the consensus gradient descent matrix by computing an average of the consensus gradient descent matrix and the additional consensus gradient descent matrix.

14. The non-transitory computer readable storage medium of claim 11 , wherein the instructions to iteratively execute the process to determine the consensus gradient descent matrix comprise instructions to, for each iteration of the process:

transmit, over the network, a first convergence indicator of the distributed computing device to a second distributed computing device of the plurality of additional distributed computing devices;

receive, over the network, a second convergence indicator of the second distributed computing device from the second distributed computing device;

update the first convergence indicator of the distributed computing device by determining a center of mass of the first convergence indicator and the second convergence indicator; and

determine whether the consensus gradient descent matrix has been obtained based on the updated first convergence indicator.

15. The non-transitory computer readable storage medium of claim 14 , wherein the instructions to determine whether the consensus gradient descent matrix has been obtained based on the updated first convergence indicator comprise instructions to determine that the first convergence indicator of the distributed computing device is within a threshold distance of a global center of mass of the first convergence indicator.

16. The non-transitory computer readable storage medium of claim 11 , wherein the instructions to generate the gradient descent matrix based on data received by the distributed computing device and a model stored on the distributed computing device comprise instructions to:

receive a plurality of pairs of training data, each pair comprising a data input and a label;

for each pair of the plurality of pairs of training data, compute a gradient vector of a plurality of gradient vectors by evaluating a partial derivative of an objective function of the model based on the pair; and

concatenate the plurality of gradient vectors to generate the gradient descent matrix.

17. The non-transitory computer readable storage medium of claim 11 , wherein the sampled gradient descent matrix represents the gradient descent matrix in a cooperative subspace common to the distributed computing devices and the plurality of additional distributed computing devices.

18. The non-transitory computer readable storage medium of claim 11 , wherein the instructions to calculate the sampled gradient descent matrix based on the gradient descent matrix and a random matrix comprise instructions to:

generate, as the random matrix, a Gaussian ensemble matrix; and

multiply the gradient descent matrix and the Gaussian ensemble matrix to generate the sampled gradient descent matrix.

19. The non-transitory computer readable storage medium of claim 11 , wherein the instructions to update the model stored on the distributed computing device based on the consensus gradient descent matrix comprise instructions to:

extract an orthogonal subspace of the consensus gradient descent matrix spanning the range of a global gradient descent matrix; and

update weights of the model based on the extracted orthogonal subspace.

20. The non-transitory computer readable storage medium of claim 11 , wherein the model is a machine learning artificial intelligence (AI) model configured to make a prediction based on one or more input signals received by the distributed computing device.