Implementing asyncronous collective operations in a multi-node processing system
A method, system, and computer program product are disclosed for implementing an asynchronous collective operation in a multi-node data processing system. In one embodiment, the method comprises sending data to a plurality of nodes in the data processing system, broadcasting a remote get to the plurality of nodes, and using this remote get to implement asynchronous collective operations on the data by the plurality of nodes. In one embodiment, each of the nodes performs only one task in the asynchronous operations, and each nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each node. In another embodiment, each of the nodes performs a plurality of tasks in said collective operations, and each task of each node sets up a base address table with an entry for a base address of a memory buffer associated with the task.
1. A method of implementing an asynchronous collective operation in a multi-node data processing system, wherein each of the nodes includes a plurality of processing units and a messaging unit to implement direct memory access to the processing units of others of the nodes, the method comprising:
using the messaging unit of one of the nodes to initiate a one sided global collective operation by injecting a remote get descriptor and broadcasting the remote get descriptor from said one of the nodes to a plurality of others of the nodes; and
said remote get descriptor initiating said global collective operation back to a specified one of the nodes wherein data are sent in an asynchronous manner in the data processing system from each of the plurality of nodes to said specified one of the nodes.
2. The method according to claim 1 , wherein each of said plurality of nodes sets up a base address table with an entry for the source memory buffer associated with said each node's contribution to the global sum.
3. The method according to claim 1 , wherein each of said plurality of nodes sets up a base address table with an entry for the address to the destination memory buffer on the target where the summed output is copied to.
4. The method according to claim 1 where all the plurality of nodes have the same physical address for a source buffer for use in said one of the collective operations, and this address is placed once in a payload of the remote get descriptor.
5. The method according to claim 1 , where each compute node has N processes and the root node initiates N remote get operations, one for each process, to complete the one-sided reduce operation and then does a local sum to complete the one-sided reduce.
6. The method according to claim 1 , wherein said broadcasting includes sending a defined communication to said plurality of destination nodes, and injecting a remote get descriptor into said defined communication, and wherein said remote get descriptor includes a put that reduces data back to a root node during said asynchronous collective operations.
7. The method according to claim 1 , wherein each of said plurality of nodes performs only one task in said asynchronous collective operations.
8. The method according to claim 7 , wherein said broadcasting includes sending a put to each of the tasks of each of the nodes to reduce data from said each task to a root node.
9. The method according to claim 1 , wherein each of the plurality of destination nodes performs a plurality of tasks in said collective operations, and wherein each of the tasks of each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each task.
10. The method according to claim 9 , wherein
said broadcasting includes sending a communication to said plurality of destination nodes, and injecting a remote get descriptor into said communication;
said descriptor includes a put that reduces data back to a root node;
said put reduces data back to the root node form a first of the tasks performed by each of the nodes during said collective operations.
11. A system for implementing an asynchronous collective operation in a multi-node data processing system, wherein each of the nodes includes a plurality of processing units and a messaging unit to implement direct memory access to the processing units of others of the nodes, the system comprising one or more processing nodes of the data processing system configured for:
using the messaging unit of one of the nodes to initiate a one sided global collective operation by injecting a remote get descriptor and broadcasting the remote get descriptor from said one of the nodes to a plurality of others of the nodes; and
said remote get descriptor initiating said global collective operation back to a specified one of the nodes wherein data are sent in an asynchronous manner in the data processing system from each of the plurality of nodes to said specified one of the nodes.
12. The system according to claim 11 , wherein the broadcasting includes using a mechanism for said Messaging Unit to initiate a one-sided allreduce operation by injecting a remote get descriptor that is broadcast to all the compute nodes and initiates a global-sum operation back to the initiating node or an arbitrary target node.
13. The system according to claim 11 , wherein each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each node.
14. The system according to claim 11 , wherein each of the plurality of destination nodes performs a plurality of tasks in said collective operations, and wherein each of the tasks of each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each task.
15. The system according to claim 11 , wherein said broadcasting includes sending a put to each of the tasks of each of the nodes to reduce data from said each task to a root node.
16. An article of manufacture comprising:
at least one tangible computer readable device having computer readable program code logic tangibility embodied therein to execute machine instructions in one or more processing units for implementing an asynchronous collective operation in a multi-node data processing system, wherein each of the nodes includes a plurality of processing units and a messaging unit to implement direct memory access to the processing units of others of the nodes, the program code logic, when executing in said one or more processing units, performing the following:
using the messaging unit of one of the nodes to initiate a one sided global collective operation by injecting a remote get descriptor and broadcasting the remote get descriptor from said one of the nodes to a plurality of others of the nodes; and
said remote get descriptor initiating said global collective operation back to a specified one of the nodes wherein data are sent in an asynchronous manner in the data processing system from each of the plurality of nodes to said specified one of the nodes.
17. The article of manufacture according to claim 16 , wherein each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each node.
18. The article of manufacture according to claim 17 , wherein each of said plurality of nodes performs only one task in said asynchronous collective operations.
19. The article of manufacture according to claim 17 , wherein each of the plurality of destination nodes performs a plurality of tasks in said collective operations, and wherein each of the tasks of each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each task.
20. The article of manufacture according to claim 19 , wherein said broadcasting includes sending a put to each of the tasks of each of the nodes to reduce data from said each task to a root node.
21. A method of implementing an asynchronous collective operation in a multi-node data processing system, wherein each of the nodes includes a plurality of processing units and a messaging unit to implement direct memory access to the processing units of others of the nodes, the method comprising:
using the messaging unit of one of the nodes to initiate a one sided global collective operation by injecting generating a remote get descriptor and broadcasting the remote get descriptor from said one of the nodes to a plurality of others of the nodes; and
said remote get descriptor initiating said global collective operation back to a specified one of the nodes wherein data are sent in an asynchronous manner in the data processing system from each of the plurality of nodes to said specified one of the nodes; and wherein
said broadcasting includes sending a defined communication to said plurality of destination nodes, and injecting the remote get descriptor into said defined communication, said remote get descriptor including a put that reduces data back to said specified one of the nodes during said asynchronous collective operation.
22. The method according to claim 21 , wherein each of said plurality of nodes performs only one task in said asynchronous collective operations, and each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with said each node.
23. The method according to claim 21 , wherein each of the plurality of destination nodes performs a plurality of tasks in said collective operations, and wherein each of the tasks of each of said plurality of nodes sets up a base address table with an entry for a base address of a memory buffer associated with aid each task.
24. The method according to claim 23 , wherein said broadcasting includes sending a put to each of the tasks of each of the nodes to reduce data from said each task to the root node.
25. The method according to claim 21 , wherein said broadcasting includes said root node injecting said remote get descriptor into said defined communication.