Abstraction layers for scalable distributed machine learning
One embodiment provides for a method of transmitting data between multiple compute nodes of a distributed compute system, the method comprising creating a global view of communication operations to be performed between the multiple compute nodes of the distributed compute system, the global view created using information specific to a machine learning model associated with the distributed compute system; using the global view to determine a communication cost of the communication operations; and automatically determining a number of network endpoints for use in transmitting the data between the multiple compute nodes of the distributed compute system.
1. A system comprising:
a system memory to store a set of trainable machine learning parameters and a machine learning scaling library (MLSL) to facilitate data transmission during distributed training of a neural network, wherein the distributed training is performed using a distributed system having multiple computational nodes, wherein respective computational nodes of the multiple computational nodes comprise multiple graphics processors, and wherein the respective computational nodes host an instance of the neural network and separate graphics processors of the multiple graphics processors are used to train different portions of the neural network;
a fabric interface to enable transmission and receipt of data associated with the set of trainable machine learning parameters;
a set of general-purpose processor cores to execute instructions provided by the MLSL, the instructions to control a data transmission library, wherein the set of general-purpose processor cores are to control the data transmission library to send and receive training data via the fabric interface during a machine learning framework workflow; and
a general-purpose graphics processor of the multiple graphics processors, the general-purpose graphics processor communicably coupled to the set of general-purpose processor cores and is to perform compute operations associated with a machine learning framework workflow of the distributed training, the compute operations to generate gradient data for the trainable machine learning parameters, wherein the computational portions of the machine learning framework workflow are wrapped with application programming interface (API) calls of the MLSL that enable the multiple computational nodes to automatically exchange the gradient data as part of the machine learning framework workflow.
2. The system as in claim 1 , wherein the general-purpose graphics processor includes the fabric interface and a local memory that is shared between the fabric interface and the general-purpose graphics processor.
3. The system as in claim 2 , wherein the fabric interface includes a hardware module that is configurable via instructions within the MLSL, the hardware module to accelerate transmission of data stored in the local memory.
4. The system as in claim 3 , wherein to perform compute operations associated with the machine learning framework workflow, the general-purpose graphics processor is to perform compute operations associated with a forward compute to generate a set of activation data.
5. The system of claim 1 , wherein the general-purpose graphics processor is further to, while performing the machine learning framework workflow, automatically exchange one or more of activation data corresponding to activations of the neural network, the gradient data with respect to the activations, and gradients corresponding to the gradient data with respect to the machine learning parameters.
6. The system of claim 5 , wherein performing the machine learning framework workflow comprises performing forward propagation computation to generate a set of activation data and performing a backward propagation computation to determine the gradient with respect to a set of trainable machine learning parameters.
7. An apparatus comprising:
a general-purpose graphics processor to:
perform compute operations associated with a machine learning framework workflow; and
generate, as part of the machine learning framework workflow, gradient data for trainable machine learning parameters during distributed training of a neural network, wherein the distributed training is performed using a distributed system having multiple computational nodes, wherein respective computational nodes of the multiple computational nodes comprise multiple graphics processors, and wherein the respective computational nodes host an instance of the neural network and separate graphics processors of the multiple graphics processors are used to train different portions of the neural network;
wherein a set of general-purpose processor cores are to control a data transmission library to send and receive training data for the machine learning framework workflow of the distributed training via a fabric interface during the machine learning framework workflow of the distributed training; and
wherein the computational portions of the machine learning framework workflow are wrapped with application programming interface (API) calls of a machine learning scaling library (MLSL) that enable the multiple computational nodes to automatically exchange the gradient data as part of the machine learning framework workflow.
8. The apparatus as in claim 7 , wherein the general-purpose graphics processor comprises the fabric interface and a local memory that is shared between the fabric interface and the general-purpose graphics processor.
9. The apparatus as in claim 8 , wherein the fabric interface comprises hardware circuitry that is configurable via instructions within the MLSL, the hardware circuitry to accelerate transmission of data stored in the local memory.
10. The apparatus of claim 8 , wherein the general-purpose graphics processor is further to, while performing the machine learning framework workflow, automatically exchange one or more of activation data corresponding to activations of the neural network, the gradient data with respect to the activations, and gradients corresponding to the gradient data with respect to the machine learning parameters.
11. The apparatus of claim 10 , wherein performing the machine learning framework workflow comprises performing forward propagation computation to generate a set of activation data and performing a backward propagation computation to determine the gradient with respect to a set of trainable machine learning parameters.
12. A method comprising:
performing, by a general-purpose graphics processor, compute operations associated with a machine learning framework workflow; and
generating, by the general-purposes graphics processor as part of the machine learning framework workflow, gradient data for trainable machine learning parameters during distributed training of a neural network, wherein the distributed training is performed using a distributed system having multiple computational nodes, wherein respective computational nodes of the multiple computational nodes comprise multiple graphics processors, and wherein the respective computational nodes host an instance of the neural network and separate graphics processors of the multiple graphics processors are used to train different portions of the neural network;
wherein a set of general-purpose processor cores are to control a data transmission library to send and receive training data for the machine learning framework workflow of the distributed training via a fabric interface during the machine learning framework workflow of the distributed training; and
wherein the computational portions of the machine learning framework workflow are wrapped with application programming interface (API) calls of a machine learning scaling library (MLSL) that enable the multiple computational nodes to automatically exchange the gradient data as part of the machine learning framework workflow.
13. The method of claim 12 , wherein the general-purpose graphics processor comprises the fabric interface and a local memory that is shared between the fabric interface and the general-purpose graphics processor.
14. The method of claim 13 , wherein the fabric interface comprises hardware circuitry that is configurable via instructions within the MLSL, the hardware circuitry to accelerate transmission of data stored in the local memory.
15. The method of claim 12 , wherein the general-purpose graphics processor is further to, while performing the machine learning framework workflow, automatically exchange one or more of activation data corresponding to activations of the neural network, the gradient data with respect to the activations, and gradients corresponding to the gradient data with respect to the machine learning parameters.
16. The method of claim 12 , wherein performing the machine learning framework workflow comprises performing forward propagation computation to generate a set of activation data and performing a backward propagation computation to determine the gradient with respect to a set of trainable machine learning parameters.
17. A non-transitory machine-readable storage medium having stored thereon executable computer program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
performing, by a general-purpose graphics processor of the one or more processors, compute operations associated with a machine learning framework workflow; and
generating, by the general-purposes graphics processor as part of the machine learning framework workflow, gradient data for trainable machine learning parameters during distributed training of a neural network, wherein the distributed training is performed using a distributed system having multiple computational nodes, wherein respective computational nodes of the multiple computational nodes comprise multiple graphics processors, and wherein the respective computational nodes host an instance of the neural network and separate graphics processors of the multiple graphics processors are used to train different portions of the neural network;
wherein a set of general-purpose processor cores are to control a data transmission library to send and receive training data for the machine learning framework workflow of the distributed training via a fabric interface during the machine learning framework workflow of the distributed training; and
wherein the computational portions of the machine learning framework workflow are wrapped with application programming interface (API) calls of a machine learning scaling library (MLSL) that enable the multiple computational nodes to automatically exchange the gradient data as part of the machine learning framework workflow.
18. The non-transitory machine-readable storage medium of claim 17 , wherein the general-purpose graphics processor comprises the fabric interface and a local memory that is shared between the fabric interface and the general-purpose graphics processor, and wherein the fabric interface comprises hardware circuitry that is configurable via instructions within the MLSL, the hardware circuitry to accelerate transmission of data stored in the local memory.
19. The non-transitory machine-readable storage medium of claim 17 , wherein the general-purpose graphics processor is further to, while performing the machine learning framework workflow, automatically exchange one or more of activation data corresponding to activations of the neural network, the gradient data with respect to the activations, and gradients corresponding to the gradient data with respect to the machine learning parameters.
20. The non-transitory machine-readable storage medium of claim 19 , wherein performing the machine learning framework workflow comprises performing forward propagation computation to generate a set of activation data and performing a backward propagation computation to determine the gradient with respect to a set of trainable machine learning parameters.