Triggered operations to improve allreduce overlap

1. A machine learning system, comprising:

memory; and

logic communicatively coupled to the memory and a neural network to:

during a backward propagation phase associated with an iterative process to train the neural network, overlap execution of a first layer of the neural network with transmission of a first message, wherein the first message is associated with a first collective operation associated with a second layer of the neural network; and

during a forward propagation phase associated with the neural network, determine that a second message is to be transmitted based on a previous identification from a previous iteration of the iterative process of a number of messages that are transmittable by a third layer of the neural network during computation of a fourth layer of the neural network, wherein the second message is related to a second collective operation for the neural network, wherein the second collective operation is an operation of the third layer of the neural network that was not completed during the backward propagation phase.

2. The system of claim 1 , wherein the logic is further to:

construct a directed acyclic graph corresponding to collective operations for the neural network including the first and second collective operations; and

offload execution of the directed acyclic graph to a hardware-based message scheduler.

3. The system of claim 1 , wherein the logic is further to:

organize a set of collective operations for gradient exchange based on all layers of the neural network.

4. The system of claim 3 , wherein the logic is further to:

overlap messages for a current layer of the neural network with messages of one or more prior layers of the neural network in the backward propagation phase,

wherein the first collective operation is an Allreduce operation, and

wherein the second collective operation is an Allreduce operation.

5. The system of claim 1 , wherein the neural network comprises a deep learning neural network.

6. A semiconductor package apparatus, comprising:

one or more substrates; and

logic coupled to the one or more substrates, wherein the logic is at least partly implemented in one or more of configurable logic or fixed-functionality logic hardware, the logic coupled to the one or more substrates to:

during a backward propagation phase associated with an iterative process to train a neural network, overlap execution of a first layer of the neural network with transmission of a first message, wherein the first message is associated with a first collective operation associated with a second layer of the neural network; and

during a forward propagation phase associated with the neural network, determine that a second message is to be transmitted based on a previous identification from a previous iteration of the iterative process of a number of messages that are transmittable by a third layer of the neural network during computation of a fourth layer of the neural network, wherein the second message is related to a second collective operation for the neural network, wherein the second collective operation is an operation of the third layer of the neural network that was not completed during the backward propagation phase.

7. The apparatus of claim 6 , wherein the logic is further to:

construct a directed acyclic graph corresponding to collective operations for the neural network including the first and second collective operations; and

offload execution of the directed acyclic graph to a hardware-based message scheduler.

8. The apparatus of claim 6 , wherein the logic is further to:

organize a set of collective operations for gradient exchange based on all layers of the neural network.

9. The apparatus of claim 8 , wherein the logic is further to:

overlap messages for a current layer of the neural network with messages of one or more prior layers of the neural network in the backward propagation phase,

wherein the first collective operation is an Allreduce operation, and

wherein the second collective operation is an Allreduce operation.

10. The apparatus of claim 6 , wherein the neural network comprises a deep learning neural network.

11. The apparatus of claim 6 , wherein the logic coupled to the one or more substrates includes transistor channel regions that are positioned within the one or more substrates.

12. A method of machine learning, comprising:

during a backward propagation phase associated with a training process for training a neural network, overlapping execution of a first layer of the neural network with transmission of a first message, wherein the first message is associated with a first collective operation associated with a second layer of the neural network; and

during a forward propagation phase associated with the neural network, determine that a second message will be transmitted based on a previous identification from a previous iteration of the iterative process of a number of messages that are transmittable by a third layer of the neural network during computation of a fourth layer of the neural network, wherein the second message is related to a second collective operation for the neural network, wherein the second collective operation is an operation of the third layer of the neural network that was not completed during the backward propagation phase.

13. The method of claim 12 , further comprising:

constructing a directed acyclic graph corresponding to collective operations for the neural network including the first and second collective operations; and

offloading execution of the directed acyclic graph to a hardware-based message scheduler.

14. The method of claim 12 , further comprising:

organizing a set of collective operations for gradient exchange based on all layers of the neural network.

15. The method of claim 14 , further comprising:

overlapping messages for a current layer of the neural network with messages of one or more prior layers of the neural network in the backward propagation phase,

wherein the first collective operation is an Allreduce operation, and

wherein the second collective operation is an Allreduce operation.

16. The method of claim 12 , wherein the neural network comprises a deep learning neural network.

17. At least one non-transitory computer readable storage medium, comprising a set of instructions, which when executed by a computing device, cause the computing device to:

during a backward propagation phase associated with an iterative process to train a neural network, overlap execution of a first layer of the neural network with transmission of a first message, wherein the first message is associated with a first collective operation associated with a second layer of the neural network; and

during a forward propagation phase associated with the neural network, determine that a second message is to be transmitted based on a previous identification from a previous iteration of the iterative process of a number of messages that are transmittable by a third layer of the neural network during computation of a fourth layer of the neural network, wherein the second message is related to a second collective operation for the neural network, wherein the second collective operation is an operation of the third layer of the neural network that was not completed during the backward propagation phase.

18. The at least one non-transitory computer readable storage medium of claim 17 , comprising a further set of instructions, which when executed by the computing device, cause the computing device to:

construct a directed acyclic graph corresponding to collective operations for the neural network including the first and second collective operations; and

offload execution of the directed acyclic graph to a hardware-based message scheduler.

19. The at least one non-transitory computer readable storage medium of claim 17 , comprising a further set of instructions, which when executed by the computing device, cause the computing device to:

organize a set of collective operations for gradient exchange based on all layers of the neural network.

20. The at least one non-transitory computer readable storage medium of claim 19 , comprising a further set of instructions, which when executed by the computing device, cause the computing device to:

overlap messages for a current layer of the neural network with messages of one or more prior layers of the neural network in the backward propagation phase,

wherein the first collective operation is an Allreduce operation, and

wherein the second collective operation is an Allreduce operation.

21. The at least one non-transitory computer readable storage medium of claim 17 , wherein the neural network comprises a deep learning neural network.