Massively parallel in-network compute
Efficient scaling of in-network compute operations to large numbers of compute nodes is disclosed. Each compute node is connected to a same plurality of network compute nodes, such as compute-enabled network switches. Compute processes at the compute nodes generate local gradients or other vectors by, for instance, performing a forward pass on a neural network. Each vector comprises values for a same set of vector elements. Each network compute node is assigned to, based on the local vectors, reduce vector data for a different a subset of the vector elements. Each network compute node returns a result chunk for the elements it processed back to each of the compute nodes, whereby each compute node receives the full result vector. This configuration may, in some embodiments, reduce buffering, processing, and/or other resource requirements for the network compute node or network at large.
1. A method comprising:
generating, by a plurality of compute nodes implemented at least in part with one or more hardware processors, vector chunks having values for a common set of vector elements, the common set of vector elements being common to all vectors generated for a common distributed application, wherein the common set of vector elements includes a plurality of subsets of vector elements, wherein each vector chunk of the vector chunks has a subset of the values for a respective subset in the plurality subsets of vector elements;
sending, to a network switch, the vector chunks over a plurality of switch ports, wherein each vector chunk of the vector chunks is sent to a respective switch port of the plurality of switch ports by a respective compute node in the plurality of compute nodes; and
receiving, from the network switch by each compute node of the plurality of compute nodes, a single result chunk, the single result chunk being formed at the network switch from a subset of vector chunks that were generated by the plurality of compute nodes and sent to the network switch.
2. The method of claim 1 , further comprising: sharing the single result chunk by a plurality of compute processes located at each compute node of the plurality of compute nodes.
3. The method of claim 2 , wherein each compute node of the plurality of compute nodes comprises one or more compute entities; wherein each compute entity of the one or more compute entities hosts one or more respective processes in the plurality of compute processes.
4. The method of claim 1 , wherein the vector chunks are reduced to the single result chunk through one or more of: summation, averaging, multiplying, selecting a minimum value, or selecting a maximum value.
5. The method of claim 1 , wherein the network switch and the plurality of compute nodes collectively implement the common distributed application.
6. The method of claim 5 , wherein the common distributed application includes one or more artificial neural networks.
7. The method of claim 5 , wherein the common distributed application represents one or more of: deep learning applications, machine learning applications or artificial intelligence applications.
8. The method of claim 1 , wherein an error occurs in processing the vector chunks received from the plurality of compute nodes; wherein a message is sent to each compute nodes in the plurality of compute nodes to inform the error.
9. The method of claim 1 , wherein each compute node in the plurality of compute nodes is implemented at least in part with one or more of: central processing units, graphics processing units, tensor processing units, floating point units, hardware accelerators, or other computing processor.
10. The method of claim 1 , wherein a plurality of network switches operates with the plurality of compute nodes to reduce sets of vector chunks sent by the plurality of compute nodes; wherein the plurality of network switches includes the network switch.
11. A system comprising:
a plurality of switch ports;
a plurality of compute nodes implemented by one or more hardware processors;
wherein the system performs:
generating, by the plurality of compute nodes implemented at least in part with one or more hardware processors, vector chunks having values for a common set of vector elements, the common set of vector elements being common to all vectors generated for a common distributed application, wherein the common set of vector elements includes a plurality of subsets of vector elements, wherein each vector chunk of the vector chunks has a subset of the values for a respective subset in the plurality subsets of vector elements;
sending, to a network switch, the vector chunks over the plurality of switch ports, wherein each vector chunk of the vector chunks is sent to a respective switch port of the plurality of switch ports by a respective compute node in the plurality of compute nodes; and
receiving, from the network switch by each compute node of the plurality of compute nodes, a single result chunk, the single result chunk being formed at the network switch from a subset of vector chunks that were generated by the plurality of compute nodes and sent to the network switch.
12. The system of claim 11 , wherein the single result chunk is shared by a plurality of compute processes located at each compute node of the plurality of compute nodes.
13. The system of claim 12 , wherein each compute node of the plurality of compute nodes comprises one or more compute entities; wherein each compute entity of the one or more compute entities hosts one or more respective processes in the plurality of compute processes.
14. The system of claim 11 , wherein the vector chunks are reduced to the single result chunk through one or more of: summation, averaging, multiplying, selecting a minimum value, or selecting a maximum value.
15. The system of claim 11 , wherein the network switch and the plurality of compute nodes collectively implement the common distributed application.
16. The system of claim 15 , wherein the common distributed application includes one or more artificial neural networks.
17. The system of claim 15 , wherein the common distributed application represents one or more of: deep learning applications, machine learning applications or artificial intelligence applications.
18. The system of claim 11 , wherein an error occurs in processing the vector chunks received from the plurality of compute nodes; wherein a message is sent to each compute nodes in the plurality of compute nodes to inform the error.
19. The system of claim 11 , wherein each compute node in the plurality of compute nodes is implemented at least in part with one or more of: central processing units, graphics processing units, tensor processing units, floating point units, hardware accelerators, or other computing processor.
20. The system of claim 11 , wherein a plurality of network switches operates with the plurality of compute nodes to reduce sets of vector chunks sent by the plurality of compute nodes; wherein the plurality of network switches includes the network switch.