Message aggregation, combining and compression for efficient data communications in GPU-based clusters

1. A processing node comprising:

a processor configured to generate a plurality of network messages; and

a network operation combiner, wherein in response to an indication that one or more network messages of the plurality of network messages are combinable, the network operation combiner is configured to:

perform a combining operation on a plurality of data elements in the one or more network messages to produce a given result; and

generate a first packed network message comprising the given result in place of the plurality of data elements, wherein the first packed network message includes fewer than all of the data elements in the one or more network messages;

a data compressor configured to replace two or more network messages with a second packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more data elements in common, the second packed network message comprising an indication of the destination node and a single copy of the one or more data elements in common.

2. The processing node as recited in claim 1 , wherein the combining operation comprises any one or more of: a commutative operation, an associative operation, a maximum function, or a minimum function.

3. The processing node as recited in claim 1 , wherein the processing node further comprises a message aggregator configured to replace two or more network messages with a third packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more different data elements, the second packed network message comprising an indication of the destination node and a copy of the one or more different data elements.

4. The processing node as recited in claim 3 , wherein the message aggregator is further configured to insert in a given packed network message an indication of a number of network messages to expand from the given packed network message.

5. The processing node as recited in claim 1 , wherein the processing node further comprises a unit that includes at least the network operation combiner, wherein the unit is configured to:

monitor a total size of the plurality of network messages generated by the processor; and

responsive to determining the total size of the plurality of network messages exceeds a threshold, send an indication to the network operation combiner to process the plurality of network messages.

6. The processing node as recited in claim 1 , wherein:

the processing node further comprises a network interface configured to receive and send network messages across a network;

the operation combiner is further configured to send an indication to the network interface to send packed network message across the network instead of one or more of the plurality of network messages.

7. The processing node as recited in claim 6 , wherein in response to receiving a given packed network message, the network interface is further configured to generate two or more network messages, each with data from the received packed network message.

8. A method comprising:

generating a plurality of network messages;

wherein in response to an indication that one or more network messages of the plurality of network messages is combinable:

performing a combining operation on a plurality of data elements in the one or more network messages to produce a given result; and

generating a first packed network message comprising the given result in place of the plurality of data elements, wherein the first packed network message includes fewer than all of the data elements in the one or more network messages;

replacing two or more network messages with a second packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more data elements in common, the second packed network message comprising an indication of the destination node and a single copy of the one or more data elements in common.

9. The method as recited in claim 8 , wherein the combining operation comprises any one or more of: a commutative operation, an associative operation, a maximum function, or a minimum function.

10. The method as recited in claim 8 , further comprising replacing two or more network messages with a third packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more different data elements, the second packed network message comprising an indication of the destination node and a copy of the one or more different data elements.

11. The method as recited in claim 10 , further comprising inserting in a given packed network message an indication of a number of network messages to expand from the given packed network message.

12. The method as recited in claim 10 , further comprising sending an indication to a network interface indicating to send packed network message across a network and bypass sending one or more of the plurality of network messages used to generate the packed network messages.

13. The method as recited in claim 8 , further comprising:

monitoring a total size of the plurality of network messages generated by the processor; and

responsive to determining the total size of the plurality of network messages exceeds a threshold, sending an indication to a network operation combiner to process the plurality of network messages.

14. The method as recited in claim 13 , further comprising inserting in a given packed network message an indication of a number of network messages to expand from the given packed network message and an indication of which data to place in which of the number of network messages.

15. A computing system comprising:

a source node;

a network; and

a target node coupled to the source node via the network; and

wherein the source node is configured to:

generate a plurality of network messages;

in response to an indication that one or more network messages of the plurality of network messages is combinable:

perform a combining operation on a plurality of data elements in the one or more network messages to produce a given result; and

generate a first packed network message comprising the given result in place of the plurality of data elements, wherein the first packed network message includes fewer than all of the data elements in the one or more network messages;

replace two or more network messages with a second packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more data elements in common, the second packed network message comprising an indication of the destination node and a single copy of the one or more data elements in common.

16. The computing system as recited in claim 15 , wherein the source node is further configured to send the first packed network message across the network to the target node and bypass sending one or more of the plurality of network message.

17. The computing system as recited in claim 15 , wherein the source node is further configured to replace two or more network messages with a third packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more different data elements, the second packed network message comprising an indication of the destination node and a copy of the one or more different data elements.