In-node aggregation and disaggregation of MPI alltoall and alltoallv collectives
An MPI collective operation carried out in a fabric of network elements by transmitting MPI messages from all the initiator processes in an initiator node to designated ones of the responder processes in respective responder nodes. Respective payloads of the MPI messages are combined in a network interface device of the initiator node to form an aggregated MPI message. The aggregated MPI message is transmitted through the fabric to network interface devices of responder nodes, disaggregating the aggregated MPI message into individual messages, and distributing the individual messages to the designated responder node processes.
1. A method of communication, comprising the steps of:
executing a group of processes in a fabric of nodes comprising Network Interface Controllers (NICs), the NICs having respective NIC Communicator Controllers (NCCs);
transmitting messages from all the processes to all the processes of the group of processes via the NICs by performing a Message Passing Interface (MPI) collective operation, wherein the nodes of the fabric function concurrently as initiator nodes executing respective initiator processes and as responder nodes executing respective responder processes, and wherein the collective operation comprises transmitting MPI messages through the fabric from all the initiator processes to all of the responder processes;
with the NCCs of the initiator nodes combining respective payloads of the MPI messages to form an aggregated MPI message;
transmitting the aggregated MPI message through the fabric to the responder nodes;
in respective NCCs of the responder nodes disaggregating the aggregated MPI message into individual messages; and
distributing the individual messages to the responder processes.
2. The method according to claim 1 , wherein the aggregated MPI message has exactly one transport header that comprises a destination address of the aggregated MPI message.
3. The method according to claim 1 , wherein the MPI messages comprise respective MPI headers comprising designations of the responder processes, wherein the responder processes are referenced in an MPI communicator object.
4. The method according to claim 3 , wherein the MPI collective operation comprises forwarding by a communication library the MPI communicator object and the payloads to the NICs of the initiator nodes.
5. The method according to claim 1 , further comprising:
maintaining a communicator context in the NCCs of the initiator nodes, wherein transmitting the aggregated MPI message comprises directing the aggregated MPI message to local identifiers (LIDs) in the responder nodes according to the communicator context.
6. The method according to claim 1 , comprising forming the aggregated MPI message by assembling pointers to message data, and including respective local identifier addresses for the message data in the aggregated MPI message.
7. An apparatus of communication, comprising:
a fabric of nodes configured for executing a group of processes;
respective network interface controllers (NICs) in the nodes; and
respective NIC Communicator Controllers (NCCs) in the NICs,
wherein the nodes are operative for transmitting messages from all the processes to all the processes of the group of processes via the NICs by performing a Message Passing Interface (MPI) collective operation, wherein the nodes of the fabric function concurrently as initiator nodes executing respective initiator processes and as responder nodes executing respective responder processes, and wherein the collective operation comprises transmitting MPI messages through the fabric from all the initiator processes to all of the responder processes; and
wherein the NCCs of the initiator nodes are configured for combining respective payloads of the MPI messages to form an aggregated MPI message, and the NCCs of the responder nodes are configured for disaggregating the aggregated MPI message into individual messages, and wherein the responder nodes are operative for distributing the individual messages to the responder processes.
8. The apparatus according to claim 7 , wherein the aggregated MPI message has exactly one transport header that comprises a destination address of the aggregated MPI message.
9. The apparatus according to claim 7 , wherein the MPI messages comprise respective MPI headers containing designations of the responder processes, wherein the responder processes are referenced in an MPI communicator object.
10. The apparatus according to claim 9 , wherein the MPI collective operation comprises forwarding by a communication library the MPI communicator object and the payloads to the NCCs of the initiator nodes.
11. The apparatus according to claim 7 , wherein the first communicator controller circuitry is operative for forming the aggregated MPI message by assembling pointers to message data, and including respective local identifier addresses for the message data in the aggregated MPI message.
12. The apparatus according to claim 7 , wherein the NCCs are operative for maintaining a communicator context and transmitting the aggregated MPI message by directing the aggregated MPI message to local identifiers (LIDs) in the responder nodes according to the communicator context.