Collective communication system and methods

1. A method comprising:

providing a plurality of processes, each of said plurality of processes being configured to hold a block of data destined for others of said plurality of processes;

providing at least one instance of data repacking circuitry comprising:

receiving circuitry configured to receive at least one block of data from at least one source process of the plurality of processes;

repacking circuitry configured to repack received data in accordance with at least one destination process of the plurality of processes; and

sending circuitry configured to send the repacked data to said at least one destination process of the plurality of processes;

receiving a set of data for all-to-all data exchange, the set of data being configured as a matrix, the matrix being distributed among the plurality of processes;

transposing the data by: each of the plurality of processes sending matrix data from said process to said data repacking circuitry; and said data repacking circuitry receiving, repacking, and sending the resulting matrix data to destination processes; and

providing a control tree configured to control the plurality of processes and the repacking circuitry,

wherein the control tree is further configured to:

receive registration messages from each of the plurality of processes;

mark a given subgroup of the plurality of processes as ready for operation when registration messages have been received from all members of said given subgroup;

when a given subgroup which is a source subgroup and a corresponding subgroup which is a destination subgroup are ready for operation, pair the given source subgroup and the given destination subgroup and assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry; and

notify each said source subgroup and each said destination subgroup when operations relating to each said source subgroup and each said destination subgroup have completed.

2. The method according to claim 1 , and wherein the control tree is configured, in addition to pairing the given source subgroup and the given destination subgroup, to assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry.

3. The method according to claim 1 , and also including providing assigning circuitry other than the control tree, the assigning circuitry being configured to assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry.

4. The method according to claim 1 and wherein the control tree comprises a reduction tree.

5. The method according to claim 4 , and also including providing assigning circuitry other than the control tree, the assigning circuitry being configured to assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry.

6. Apparatus comprising:

receiving circuitry configured to receive at least one block of data from at least one source process of a plurality of processes, each of said plurality of processes being configured to hold a block of data destined for others of said plurality of processes;

at least one instance of data repacking circuitry configured to repack received data in accordance with at least one destination process of the plurality of processes; and

sending circuitry configured to send the repacked data to said at least one destination process of the plurality of processes,

the apparatus being configured to receive a set of data for all-to-all data exchange, the set of data being configured as a matrix, the matrix being distributed among the plurality of processes, and

the apparatus being further configured to transpose the data by: receiving, from each of the plurality of processes, matrix data from said process at the repacking circuitry; and the data repacking circuitry receiving, repacking, and sending the resulting matrix data to destination processes,

the apparatus comprising a control tree configured to control the plurality of processes and the repacking circuitry, and

wherein the control tree is further configured to:

receive registration messages from each of the plurality of processes;

mark a given subgroup of the plurality of processes as ready for operation when registration messages have been received from all members of said given subgroup;

when a given subgroup which is a source subgroup and a corresponding subgroup which is a destination subgroup are ready for operation, pair the given source subgroup and the given destination subgroup and assign the given source subgroup and the given destination subgroup to an instance of data repacking circuitry; and

notify each said source subgroup and each said destination subgroup when operations relating to each said source subgroup and each said destination subgroup have completed.

7. Apparatus according to claim 6 , and wherein the control tree is configured, in addition to pairing the given source subgroup and the given destination subgroup, to assign the given source subgroup and the given destination subgroup to a given instance of data repacking circuitry.

8. Apparatus according to claim 6 , and also comprising assigning circuitry other than the control tree, the assigning circuitry being configured to assign the given source subgroup and the given destination subgroup to a given instance of data repacking circuitry.

9. Apparatus according to claim 8 and wherein the control tree comprises a reduction tree.

10. Apparatus according to claim 6 and wherein the control tree comprises a reduction tree.