Communication optimizations for distributed machine learning
Embodiments described herein provide a system to configure distributed training of a neural network, the system comprising memory to store a library to facilitate data transmission during distributed training of the neural network; a network interface to enable transmission and receipt of configuration data associated with a set of worker nodes, the worker nodes configured to perform distributed training of the neural network; and a processor to execute instructions provided by the library, the instructions to cause the processor to create one or more groups of the worker nodes, the one or more groups of worker nodes to be created based on a communication pattern for messages to be transmitted between the worker nodes during distributed training of the neural network.
1. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
tracking, at a machine learning framework, communication behavior of applications communicating on a machine learning platform via quality of service (QoS) counters exposed by one or more of the one or more processors of the machine learning platform;
determining a relative share of intra-chassis and inter-chassis communication bandwidth consumed by each application managed by the machine learning framework;
determining congestion points within intra-chassis and inter-chassis communication links; and
adjusting an application communication schedule based on a relative share of communication bandwidth and congestion points within intra-chassis and inter-chassis communication links, wherein adjusting the application communication schedule includes scheduling an inter-chassis message in place of an intra-chassis message in response to determining existence of intra-chassis congestion.
2. The non-transitory machine-readable medium as in claim 1 , the operations additionally comprising dynamically adjusting cache QoS models of the one or more processors to prioritize data associated with a latency sensitive application.
3. The non-transitory machine-readable medium as in claim 2 , wherein prioritizing data associated with the latency sensitive application includes lowering a probability of eviction for the data.
4. The non-transitory machine-readable medium as in claim 1 , the operations additionally comprising adjusting a communication route for an application message between the one or more processors of the machine learning platform.
5. The non-transitory machine-readable medium as in claim 4 , wherein adjusting the communication route for an application message between the one or more processors of the machine learning platform includes determining the existence of intra-chassis congestion and adjusting the communication route for the application message based on the intra-chassis congestion.
6. The non-transitory machine-readable medium as in claim 1 , the operations additionally comprising scheduling an intra-chassis message in place of an inter-chassis message in response to determining existence of inter-chassis congestion.
7. A method comprising:
tracking, at a machine learning framework, communication behavior of applications communicating on a machine learning platform via quality of service (QoS) counters exposed by one or more processors of the machine learning platform;
determining a relative share of intra-chassis and inter-chassis communication bandwidth consumed by each application managed by the machine learning framework;
determining congestion points within intra-chassis and inter-chassis communication links; and
adjusting an application communication schedule based on a relative share of communication bandwidth and congestion points within intra-chassis and inter-chassis communication links, wherein adjusting the application communication schedule includes scheduling an inter-chassis message in place of an intra-chassis message in response to determining existence of intra-chassis congestion.
8. The method as in claim 7 , further comprising dynamically adjusting cache QoS models of the one or more processors to prioritize data associated with a latency sensitive application.
9. The method as in claim 8 , wherein prioritizing data associated with the latency sensitive application includes lowering a probability of eviction for the data.
10. The method as in claim 7 , further comprising adjusting a communication route for an application message between the one or more processors of the machine learning platform.
11. The method as in claim 10 , wherein adjusting the communication route for an application message between the one or more processors of the machine learning platform includes determining the existence of intra-chassis congestion and adjusting the communication route for the application message based on the intra-chassis congestion.
12. The method as in claim 7 , further comprising scheduling an intra-chassis message in place of an inter-chassis message in response to determining existence of inter-chassis congestion.
13. A data processing system comprising:
a memory device; and
one or more processors coupled with the memory device, wherein the one or more processors are configured to execute instructions stored on the memory device and the instructions cause the one or more processors to perform operations comprising:
tracking, at a machine learning framework, communication behavior of applications communicating on a machine learning platform via quality of service (QoS) counters exposed by one or more of the one or more processors of the machine learning platform;
determining a relative share of intra-chassis and inter-chassis communication bandwidth consumed by each application managed by the machine learning framework;
determining congestion points within intra-chassis and inter-chassis communication links; and
adjusting an application communication schedule based on a relative share of communication bandwidth and congestion points within intra-chassis and inter-chassis communication links, wherein adjusting the application communication schedule includes scheduling an inter-chassis message in place of an intra-chassis message in response to determining existence of intra-chassis congestion.
14. The data processing system as in claim 13 , further comprising dynamically adjusting cache QoS models of the one or more processors to prioritize data associated with a latency sensitive application.
15. The data processing system as in claim 14 , wherein prioritizing data associated with the latency sensitive application includes lowering a probability of eviction for the data.
16. The data processing system as in claim 13 , further comprising adjusting a communication route for an application message between the one or more processors of the machine learning platform.
17. The data processing system as in claim 16 , wherein adjusting the communication route for an application message between the one or more processors of the machine learning platform includes determining the existence of intra-chassis congestion and adjusting the communication route for the application message based on the intra-chassis congestion.
18. The data processing system as in claim 13 , further comprising scheduling an intra-chassis message in place of an inter-chassis message in response to determining existence of inter-chassis congestion.
19. The data processing system as in claim 13 , further comprising a first chassis including a first set of graphics processors of the one or more processors and a second chassis including a second set of graphics processors of the one or more processors, wherein the inter-chassis communication links enable communication between the first chassis and the second chassis.