Communication optimizations for distributed machine learning
Embodiments described herein provide an apparatus comprising an interconnect switch configured to couple with a plurality of graphics processors via a plurality of point-to-point interconnects and one or more processors including a graphics processor coupled with the interconnect switch via a point-to-point interconnect of the plurality of point-to-point interconnects.
1. An apparatus comprising:
an interconnect switch configured to couple with a plurality of graphics processors via a plurality of point-to-point interconnects; and
one or more processors including a graphics processor, the graphics processor coupled with the interconnect switch via a point-to-point interconnect of the plurality of point-to-point interconnects, the graphics processor comprising:
a cluster of graphics multiprocessors configured for single instruction multiple thread (SIMT) operation, the cluster of graphics multiprocessors interconnected via a data interconnect and configured to exchange data via the data interconnect, the cluster of graphics multiprocessors including a graphics multiprocessor configured to:
receive data associated with a first thread group to be executed via the graphics multiprocessor during execution of operations associated with a second thread group, the data to be received via a point-to-point interconnect coupled with the interconnect switch in association with a communication pattern for messages to be transmitted between worker nodes of a first group of worker nodes configured to perform distributed training of a neural network; and
transmit data processed by the second thread group via the point-to-point interconnect coupled with the interconnect switch during execution of operations the first thread group.
2. The apparatus of claim 1 , wherein the communication pattern is an allreduce pattern.
3. The apparatus of claim 1 , wherein worker nodes in the first group of worker nodes are grouped according to topological locality and configured to perform allreduce synchronization of data between the worker nodes.
4. The apparatus of claim 3 , wherein the data associated with the first thread group includes parameter data.
5. The apparatus of claim 4 , wherein the first thread group is to transmit updated parameter data to a parameter server via the interconnect switch via the point-to-point interconnect.
6. The apparatus of claim 5 , the parameter server to synchronize the updated parameter data with second group of worker nodes, the second group of worker nodes grouped according to topological locality.
7. The apparatus of claim 6 , further comprising a second graphics processor of the plurality of graphics processors coupled with the interconnect switch via a second point-to-point interconnect, the second graphics processor configured as a worker node in the first group of worker nodes.
8. A method comprising:
transmitting configuration data associated with a set of worker nodes of a distributed training system configured to perform distributed training of a neural network, each worker node including a graphics processor configured to perform compute operations associated with a machine learning framework workflow, the set of worker nodes interconnected via an interconnect switch configured to couple with a plurality of graphics processors via a plurality of point-to-point interconnects and enable communication between the plurality of graphics processors;
creating a group of worker nodes based on a communication pattern for messages to be transmitted between during the distributed training of the neural network, wherein worker nodes of the group of worker nodes are automatically grouped according to locality; and
facilitating transmission between worker nodes in the group of worker nodes according to the communication pattern.
9. The method of claim 8 , further comprising configuring the interconnect switch to accelerate transmission of data between the worker nodes according to the communication pattern.
10. The method of claim 8 , further comprising transparently adjusting communication paths between the worker nodes based on the communication pattern.
11. The method of claim 10 , at least a portion of the worker nodes additionally interconnected via point-to-point interconnects between interconnected worker nodes.
12. The method of claim 11 , further comprising transparently adjusting communication paths between the worker nodes based on communication congestion points between worker nodes of the group of worker nodes.
13. The method as in claim 8 , further comprising performing an all-reduce synchronization of parameter data between the worker nodes within the group of worker nodes via the interconnect switch.
14. A data processing system comprising:
a host processor;
an interconnect switch coupled with the host processor, the interconnect switch configured to couple with a plurality of graphics processors via a plurality of point-to-point interconnects; and
one or more processors including a graphics processor, the graphics processor coupled with the interconnect switch via a point-to-point interconnect of the plurality of point-to-point interconnects, the graphics processor comprising:
a cluster of graphics multiprocessors configured for single instruction multiple thread (SIMT) operation, the cluster of graphics multiprocessors interconnected via a data interconnect and configured to exchange data via the data interconnect, the cluster of graphics multiprocessors including a graphics multiprocessor configured to:
receive data associated with a first thread group to be executed via the graphics multiprocessor during execution of operations associated with a second thread group, the data to be received via a point-to-point interconnect coupled with the interconnect switch in association with a communication pattern for messages to be transmitted between worker nodes of a first group of worker nodes configured to perform distributed training of a neural network; and
transmit data processed by the second thread group via the point-to-point interconnect coupled with the interconnect switch during execution of operations the first thread group.
15. The data processing system of claim 14 , wherein the communication pattern is an allreduce pattern.
16. The data processing system of claim 14 , wherein worker nodes in the first group of worker nodes are grouped according to topological locality and configured to perform allreduce synchronization of data between the worker nodes.
17. The data processing system of claim 16 , wherein the data associated with the first thread group includes parameter data.
18. The data processing system of claim 17 , wherein the first thread group is to transmit updated parameter data to a parameter server via the interconnect switch via the point-to-point interconnect.
19. The data processing system of claim 18 , the parameter server to synchronize the updated parameter data with second group of worker nodes, the second group of worker nodes grouped according to topological locality.
20. The data processing system of claim 19 , further comprising a second graphics processor of the plurality of graphics processors coupled with the interconnect switch via a second point-to-point interconnect, the second graphics processor configured as a worker node in the first group of worker nodes.