Network computer with two embedded rings
A computer comprising a plurality of interconnected processing nodes arranged in a configuration in which multiple layers of interconnected nodes are arranged along an axis, each layer comprising at least four processing nodes connected in a non-axial ring by at least respective intralayer link between each pair of neighbouring processing nodes, wherein each of the at least four processing nodes in each layer is connected to a respective corresponding node in one or more adjacent layer by a respective interlayer link, the computer being programmed to provide in the configuration two embedded one dimensional paths and to transmit data around each of the two embedded one dimensional paths, each embedded one dimensional path using all processing nodes of the computer in such a manner that the two embedded one dimensional paths operate simultaneously without sharing links.
1. A computer comprising:
a plurality of interconnected routing nodes arranged in a configuration in which multiple layers of interconnected nodes are arranged along an axis, each layer comprising at least four routing nodes connected in a non-axial ring by at least one respective intralayer links between each pair of neighbouring routing nodes,
wherein each of the at least four routing nodes in each layer is connected to a respective corresponding node in one or more adjacent layer by a respective interlayer link, the computer being programmed to provide in the configuration two embedded one-dimensional paths and to transmit data around each of the two embedded one-dimensional paths, each embedded one-dimensional path using all routing nodes of the computer in such a manner that the two embedded one-dimensional paths operate simultaneously without sharing links,
wherein the multiple layers comprise first and second endmost layers and at least one intermediate layer between the first and second endmost layers, wherein each routing node in the first endmost layer is connected to a non-neighbouring node in the first endmost layer in addition to its neighbouring node, and each routing node in the second endmost layer is connected to a non-neighbouring node in the second endmost layer in addition to its neighbouring node,
wherein each routing node comprises a memory attachment and routing chip which is connected to one or more processor cores which comprise processor circuitry configured to process the data.
2. The computer of claim 1 , wherein the configuration is a toroid configuration in which respective connected corresponding nodes of the multiple layers form at least four axial rings.
3. The computer of claim 1 wherein at least one of the interlayer links and the intralayer links comprises switching circuitry operable to connect one of the routing nodes selectively to one of multiple other routing nodes.
4. The computer of claim 1 , wherein each routing node is configured to output data on its respective intralayer links and interlayer links with a same bandwidth utilisation on each of the intralayer links and the interlayer links of the routing node.
5. The computer of claim 1 , wherein each layer of the multiple layers has exactly four nodes.
6. The computer of claim 1 which comprises a number of layers arranged along the axis which is greater than a number of routing nodes in each layer.
7. The computer of claim 1 which comprises a number of layers arranged along the axis which is the same as a number of nodes in each layer.
8. The computer of claim 1 wherein the intralayer links and the interlayer links comprise fixed connections between the routing nodes.
9. The computer of claim 1 wherein at least one of the interlayer links of routing nodes in the first endmost layer comprise switching circuitry operable to disconnect a first routing node from its neighbouring node in the first endmost layer and connect the first routing node to a corresponding node in the second endmost layer.
10. The computer of claim 1 wherein each embedded one-dimensional path comprises alternating sequences of one of the interlayer links and one of the intralayer links.
11. The computer of claim 1 in which each one-dimensional embedded path comprises a sequence of routing nodes which are visited in a direction in each layer which is the same in all layers within each one-dimensional path.
12. The computer of claim 1 in which each one-dimensional embedded path comprises a sequence of processing nodes which are visited in a direction in each layer which is different in successive layers within each one-dimensional path.
13. The computer of claim 1 wherein each routing node comprises a cluster comprising multiple memory attachment and routing chips each connected to multiple processor cores.
14. The computer of claim 1 , wherein each routing node is programmed to divide a respective partial vector of that routing node into fragments and to transmit the data in a form of successive fragments around each embedded one-dimensional path.
15. The computer of claim 14 , which is programmed to operate each path as a set of logical rings, wherein the successive fragments are transmitted around each logical ring in simultaneous transmission steps.
16. The computer of claim 14 , wherein each processing node is configured to output a respective fragment on each of two links simultaneously, wherein the fragment output on each of the links has approximately a same size.
17. The computer of claim 1 , wherein each processor core is configured to reduce multiple incoming fragments with multiple respective corresponding locally stored fragments.
18. The computer of claim 17 , wherein each processing node is configured to transmit fully reduced fragments on each of its intralayer and interlayer links simultaneously in an Allgather phase of an Allreduce collective.
19. A method of generating a set of programs to be executed in parallel on a computer comprising a plurality of routing nodes connected in a configuration with multiple layers arranged along an axis, each layer comprising at least four routing nodes connected in a non-axial ring by a respective intralayer link between each pair of neighbouring routing nodes, wherein routing nodes each comprise a memory attachment and routing chip which is connected to one or more processor cores which comprise processor circuitry configured to process the data and the routing nodes in each layer are connected to respective corresponding routing nodes in each adjacent layer by an interlayer link, the method comprising:
generating a first data transmission instruction for a first program to define a first data transmission stage in which data is transmitted from a first node executing the first program, wherein the first data transmission instruction comprises a first link identifier which defines a first outgoing link on which data is to be transmitted from the first node in the first data transmission stage;
generating a second data transmission instruction for a second program to define a second data transmission stage in which data is transmitted from a second node executing the second program, wherein the second data transmission instruction comprises a second link identifier which defines a second outgoing link on which data is to be transmitted from the second node in the second data transmission stage; and
determining the first link identifier and the second link identifier transmit data around each of two embedded one-dimensional paths provided by the configuration, each embedded one-dimensional path using all routing nodes of the computer in such a manner that the embedded one-dimensional paths operate simultaneously without sharing links, wherein the first program comprises an additional instruction to deactivate any of its interlayer and intralayer links which are not used in data transmission.
20. The method of claim 19 wherein the first program comprises an additional instruction to deactivate any of its interlayer and intralayer links which are not used in data transmission.
21. The method of claim 19 , wherein the first program comprises an additional instruction to divide a respective partial vector of the first node into fragments and to transmit the data in a form of successive fragments over the first outgoing link.
22. The method of claim 21 , further comprising transmitting the data from the first node in data transmission steps wherein the first outgoing link is utilised with a same bandwidth as a further outgoing link of the first node in each data transmission step.