Network-centric architecture and algorithms to accelerate distributed training of neural networks

1 . A distributed network comprising:

a first group of computing devices comprising computing devices circularly and sequentially coupled from a first computing device, to at least a second computing device, and to a final computing device coupled to the first computing device, wherein respective ones of the first group of computing devices are to sequentially aggregate gradient values from the first computing device to the final computing device, wherein the gradient values are data for training a neural network model and stored in each computing device of the first group of computing devices before aggregation begins, and wherein, to sequentially aggregate the gradient values, each computing device of the first group of computing devices, except the first computing device, is to:

combine locally stored gradient values with a partial aggregate of gradient values received from a previous device of the first group of computing devices; and

store the combined gradient values locally;

wherein a first aggregate of the gradient values is buffered at the first computing device and the first computing device is to transfer the first aggregate of the gradient values to a second group of computing devices communicatively coupled to the first group of computing devices; and

the second group of computing devices comprising computing devices circularly and sequentially coupled from a second first computing device to a second final computing device, wherein the second final computing device is also coupled to the second first computing device, and wherein the second group of computing devices is to generate, from the first aggregate of the gradient values, a second aggregate of gradient values to be used to train the neural network model.

2 . The distributed network of claim 1 , wherein the second computing device of the first group of computing devices is to receive new gradient values concurrently with transferring a first partial aggregate of gradient values to a third computing device of the first group of computing devices, and wherein the new gradient values are received from the first computing device.

3 . The distributed network of claim 1 , wherein, to sequentially aggregate the gradient values:

the first computing device is to transfer first gradient values to the second computing device, of the first group of computing devices, coupled to the first computing device;

the second computing device is to aggregate second gradient values with the first gradient values to generate a first partial aggregate of gradient values and transfer the first partial aggregate of gradient values to a third computing device, of the first group of computing devices, coupled to the second computing device;

the third computing device is to aggregate third gradient values with the first partial aggregate of gradient values to generate a second partial aggregate of gradient values and transfer the second partial aggregate of gradient values to at least one subsequent computing device, of the first group of computing devices, coupled to the third computing device;

the at least one subsequent computing device is to aggregate additional gradient values with the second partial aggregate of gradient values to generate a final partial aggregate of gradient values and transfer the final partial aggregate of gradient values to the final computing device, wherein the final computing device is coupled to the at least one subsequent computing device; and

the final computing device is to aggregate final gradient values with the final partial aggregate of gradient values to generate the first aggregate of gradient values.

4 . The distributed network of claim 1 , wherein, to generate the second aggregate of gradient values:

the second first computing device is to transfer first gradient values to a second computing device, of the second group of computing devices, coupled to the second first computing device;

the second computing device is to aggregate second gradient values with the first gradient values to generate a first partial aggregate of gradient values and transfer the first partial aggregate of gradient values to a third computing device, of the second group of computing devices, coupled to the second computing device;

the third computing device is to aggregate third gradient values with the first partial aggregate of gradient values to generate a second partial aggregate of gradient values and transfer the second partial aggregate of gradient values to at least one subsequent computing device, of the second group of computing devices, coupled to the third computing device;

the at least one subsequent computing device is to aggregate additional gradient values with the second partial aggregate of gradient values to generate a final partial aggregate of gradient values and transfer the final partial aggregate of gradient values to the second final computing device, wherein the second final computing device is coupled to the at least one subsequent computing device; and

the second final computing device is to aggregate final gradient values with the final partial aggregate of gradient values to generate the second aggregate of gradient values.

5 . The distributed network of claim 1 , further comprising a third group of computing devices circularly and sequentially coupled from a third first computing device to a third final computing device that is coupled to the third first computing device, wherein:

the second group of computing devices is to transfer the second aggregate of gradient values to the third group of computing device;

respective ones of the third group of computing devices are to sequentially aggregate gradient values, wherein the aggregate gradient values are stored in each computing device of the third group of computing devices, starting with the second aggregate of gradient values, from the third first computing device to the third final computing device to obtain a third aggregate of gradient values at the final computing device; and

the third first computing device is to transfer the third aggregate of gradient values to a fourth group of computing devices communicatively coupled to the third group of computing devices, wherein the third aggregate of gradient values is to be used to train the neural network model.

6 . The distributed network of claim 5 , further comprising a fourth group of computing devices circularly and sequentially coupled from a fourth first computing device to a fourth final computing device coupled to the fourth first computing device, wherein:

the third group of computing devices is to transfer the third aggregate of gradient values to the fourth group of computing device;

respective ones of the fourth group of computing devices are to sequentially aggregate gradient values, wherein the aggregate gradient values are stored in each computing device of the fourth group of computing devices, starting with the third aggregate of gradient values, from the fourth first computing device to the fourth final computing device to obtain a fourth aggregate of gradient values at the final computing device; and

the fourth first computing device is to transfer the fourth aggregate of gradient values to a fifth group of computing devices communicatively coupled to the fourth group of computing devices, wherein the fourth aggregate of gradient values is to be used to train the neural network model.

7 . The distributed network of claim 1 , wherein, to perform compression during training, each computing device of at least some of the first group of computing devices is to:

evenly partition a vector of gradient values into multiple numbered blocks of gradient values;

receive a first numbered block, of the multiple numbered blocks of gradient values, from a first neighbor computing device;

perform sum-reduction on an own first numbered block and the first numbered block of the first neighbor computing device, to generate an updated first numbered block; and

transfer the updated first numbered block to a second neighbor computing device for further sum-reduction in a cascaded manner using, at each iteration of sum-reduction, a received updated numbered block and an own updated numbered block, to obtain a different subset of the first aggregate of gradient values.

8 . The distributed network of claim 7 , wherein each computing device of the at least some of the first group of computing devices is further to:

propagate a first subset of the first aggregate of gradient values to a different-numbered computing device of the first group of computing devices; and

simultaneously receive a second subset of the first aggregate of gradient values from a different-numbered computing device of the first group of computing devices.

9 . A method of operating a distributed network of a first group of computing devices comprising computing devices circularly and sequentially coupled from a first computing device, to at least a second computing device, and to a final computing device coupled to the first computing device, and a second group of computing devices comprising computing devices circularly and sequentially coupled from a second first computing device to a second final computing device, wherein the second final computing device is also coupled to the second first computing device, the method comprising:

sequentially aggregating, by respective ones of the first group of computing devices, gradient values from the first computing device to the final computing device, wherein the gradient values are data for training a neural network model and stored in each computing device of the first group of computing devices before the aggregating begins, and wherein the sequentially aggregating the gradient values comprises each computing device of the first group of computing devices, except the first computing device:

combining locally stored gradient values with a partial aggregate of gradient values received from a previous device of the first group of computing devices; and

storing the combined gradient values locally;

wherein storing the aggregate gradient values in each computing device of the first group of computing devices results in a first aggregate of gradient values being buffered at the first computing device;

transferring, by the first computing device, the first aggregate of gradient values to the second group of computing devices; and

generating, by the second group of computing devices, from the first aggregate of gradient values, a second aggregate of gradient values to be used to train the neural network model.

10 . The method of claim 9 , further comprising receiving, by the second computing device of the first group of computing devices, new gradient values concurrently with transferring a first partial aggregate of gradient values to a third computing device of the first group of computing devices, wherein the new gradient values are received from the first computing device.

11 . The method of claim 9 , wherein, to sequentially aggregate the gradient values, the method further comprising:

transferring, by the first computing device, first gradient values to the second computing device, of the first group of computing devices, coupled to the first computing device;

aggregating, by the second computing device, second gradient values with the first gradient values to generate a first partial aggregate of gradient values;

transferring, by the second computing device, the first partial aggregate of gradient values to at least one subsequent computing device, of the first group of computing devices, coupled to the second computing device;

aggregating, by the at least one subsequent computing device, additional gradient values with the first partial aggregate of gradient values to generate a final partial aggregate of gradient values;

transferring, by the at least one subsequent computing device, the final partial aggregate of gradient values to the final computing device, wherein the final computing device is coupled to the at least one subsequent computing device; and

aggregating, by the final computing device, final gradient values with the final partial aggregate of gradient values to generate the first aggregate of gradient values.

12 . The method of claim 9 , wherein generating the second aggregate of gradient values comprises:

transferring, by the second first computing device, first gradient values to a second computing device, of the second group of computing devices, coupled to the second first computing device;

aggregating, by the second computing device, second gradient values with the first gradient values to generate a first partial aggregate of gradient values;

transferring, by the second computing device, the first partial aggregate of gradient values to at least one subsequent computing device, of the second group of computing devices, coupled to the second computing device;

aggregating, by the at least one subsequent computing device, additional gradient values with the first partial aggregate of gradient values to generate a final partial aggregate of gradient values;

transferring, by the at least one subsequent computing device, the final partial aggregate of gradient values to the second final computing device, wherein the second final computing device is coupled to the at least one subsequent computing device; and

aggregating, by the second final computing device, final gradient values with the final partial aggregate of gradient values to generate the second aggregate of gradient values.

13 . The method claim 12 , wherein the distributed network further comprises a third group of computing devices circularly and sequentially coupled from a third first computing device to a third final computing device coupled to the third first computing device, and a fourth group of computing devices communicatively coupled to the third group of computing devices, the method further comprising:

transferring, by the second group of computing devices, the second aggregate of gradient values to the third group of computing device;

sequentially aggregating, by respective ones of the third group of computing devices, aggregate gradient values, wherein the aggregate gradient values are stored in each computing device of the third group of computing devices, starting with the second aggregate of gradient values, from the third first computing device to the third final computing device to obtain a third aggregate of gradient values at the final computing device; and

transferring, by the third final computing device, the third aggregate of gradient values to the fourth group of computing devices, wherein the third aggregate of gradient values is to be used to train the neural network model.

14 . The method claim 13 , wherein the distributed network further comprises a fourth group of computing devices circularly and sequentially coupled from a fourth first computing device to a fourth final computing device coupled to the fourth first computing device, and a fifth group of computing devices communicatively coupled to the fourth group of computing devices, the method further comprising:

transferring, by the third group of computing devices, the third aggregate of gradient values to the fourth group of computing device;

sequentially aggregating, by respective ones of the fourth group of computing devices, gradient values, wherein the gradient values are stored in each computing device of the fourth group of computing devices, starting with the third aggregate of gradient values, from the fourth first computing device to the fourth final computing device to obtain a fourth aggregate of gradient values at the final computing device; and

transferring, by the fourth first computing device, the fourth aggregate of gradient values to the fifth group of computing devices, wherein the fourth aggregate of gradient values is to be used to train the neural network model.

15 . The method of claim 9 , further comprising performing compression during training, wherein the performing comprises, by each computing device of at least some of the first group of computing devices:

evenly partitioning a vector of gradient values into multiple numbered blocks of gradient values;

receiving a first numbered block, of the multiple numbered blocks of gradient values, from a first neighbor computing device;

performing sum-reduction on an own first numbered block and the first numbered block of the first neighbor computing device, to generate an updated first numbered block; and

transferring the updated first numbered block to a second neighbor computing device for further sum-reduction in a cascaded manner using, at each iteration of sum-reduction, a received updated numbered block and an own updated numbered block, to obtain a different subset of the first aggregate of gradient values.

16 . The method of claim 15 , further comprising, by each computing device of the at least some of the first group of computing devices:

propagating a first subset of the first aggregate of gradient values to a different-numbered computing device of the first group of computing devices; and

simultaneously receiving a second subset of the first aggregate of gradient values from a different-numbered computing device of the first group of computing devices.