Collective communication system and methods
A method includes providing a plurality of processes interconnected by a network, each of the plurality of processes being configured to hold a block of data destined for others of the plurality of processes. A set of data for all-to-all data exchange is received from one or more of the processes. The set of data is configured as a plurality of blocks of data in a matrix as matrix data, the matrix being distributed among the plurality of processes. The matrix data is transposed by changing the position of selected blocks of data of the plurality of blocks of data relative to the other blocks of data of the plurality of the blocks of data, without changing the structure of each of the blocks of data. The transposed matrix data is over the network and is then received, repacked, and conveyed to destination processes.
1. A method comprising:
providing a plurality of processes interconnected by a network, each of the plurality of processes being configured to hold a block of data destined for others of the plurality of processes;
receiving from one or more of the processes a set of data for all-to-all data exchange, the set of data being configured as a plurality of blocks of data in a matrix as matrix data, the matrix being distributed among the plurality of processes;
transposing the matrix data by changing the position of selected blocks of data of the plurality of blocks of data relative to the other blocks of data of the plurality of the blocks of data, without changing the structure of each of the blocks of data;
sending the transposed matrix data over the network; and
receiving and repacking the transposed matrix data from the network, and conveying the repacked matrix data to destination processes.
2. The method according to claim 1 , and also comprising providing a control tree configured to control the plurality of processes and repacking the transposed matrix data.
3. The method according to claim 2 , and wherein the control tree is further configured to:
receive registration messages from each of the plurality of processes;
mark a given subgroup of the plurality of processes as ready for operation when registration messages have been received from all members of said given subgroup;
when a given subgroup which is a source subgroup and a corresponding subgroup which is a destination subgroup are ready for operation, pair the given source subgroup and the given destination subgroup; and
notify each said source subgroup and each said destination subgroup when operations relating to each said source subgroup and each said destination subgroup have completed.
4. The method according to claim 3 , and wherein the control tree is configured, in addition to pairing the given source subgroup and the given destination subgroup, to assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry.
5. The method according to claim 2 and wherein the control tree comprises a reduction tree.
6. The method according to claim 2 , and also comprising providing assigning circuitry other than the control tree, the assigning circuitry being configured to assign a given source subgroup and a given destination subgroup to an instance of data repacking circuitry.
7. A computer software product for use in conjunction with a plurality of processes interconnected by a network, each of the plurality of processes being configured to hold a block of data destined for others of the plurality of processes, the product comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by one or more processors, cause the one or more processors:
to receive from one or more of the processes a set of data for all-to-all data exchange, the set of data being configured as a plurality of blocks of data in a matrix as matrix data, the matrix being distributed among the plurality of processes;
to transpose the matrix data by changing the position of selected blocks of data of the plurality of blocks of data relative to the other blocks of data of the plurality of the blocks of data, without changing the structure of each of the blocks of data;
to send the transposed matrix data over the network; and
to receive and repack the transposed matrix data from the network, and to convey the repacked matrix data to destination processes.
8. The product according to claim 7 , wherein the instructions cause the one or more processors to provide a control tree configured to control the plurality of processes and repacking the transposed matrix data.
9. The product according to claim 8 , and wherein the control tree is further configured to:
receive registration messages from each of the plurality of processes;
mark a given subgroup of the plurality of processes as ready for operation when registration messages have been received from all members of said given subgroup;
when a given subgroup which is a source subgroup and a corresponding subgroup which is a destination subgroup are ready for operation, pair the given source subgroup and the given destination subgroup; and
notify each said source subgroup and each said destination subgroup when operations relating to each said source subgroup and each said destination subgroup have completed.
10. The product according to claim 9 , and wherein the control tree is configured, in addition to pairing the given source subgroup and the given destination subgroup, to assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry.
11. The product according to claim 8 , and wherein the control tree comprises a reduction tree.
12. The product according to claim 8 , wherein the instructions cause the one or more processes to assign a given source subgroup and a given destination subgroup to an instance of data repacking circuitry.