Telemetry-based load-balanced fine-grained adaptive routing in high-performance system interconnect
A switch is provided for routing packets in an interconnection network. The switch includes egress ports to transmit packets, and ingress ports to receive packets. The switch also includes a buffer capacity circuit configured to obtain local buffer capacity for buffers configured to buffer packets transmitted via the switch. The switch also includes a telemetry circuit configured to receive telemetry flow control units from next switches coupled to the switch. Each telemetry flow control unit corresponds to buffer capacity at a respective next switch. The switch also includes a network capacity circuit configured to compute network capacity for transmitting packets to a destination based on the telemetry flow control units and the local buffer capacity. The switch also includes a routing circuit configured to receive packets via the ingress ports, and route the packets to the destination, via the egress ports, with bandwidth proportional to the network capacity.
1. A switch for routing packets in an interconnection network, the switch comprising:
a plurality of egress ports to transmit packets;
one or more ingress ports to receive packets;
a buffer capacity circuit configured to obtain local buffer capacity for a plurality of buffers configured to buffer packets transmitted across the interconnection network via the switch;
a telemetry circuit configured to (i) receive a plurality of telemetry flits from a plurality of next switches coupled to the switch, wherein each telemetry flow control units flit corresponds to buffer capacity at a respective next switch, (ii) obtain a telemetry tree that includes the switch as a root and the plurality of next switches coupled to the switch, as nodes of the telemetry tree, according to a topology of the interconnection network, and (iii) generate a plurality of new telemetry flits based on the buffer capacity, the plurality of telemetry flits, and the telemetry tree;
a network capacity circuit configured to compute network capacity for transmitting packets to a destination, via the plurality of egress ports, based on the plurality of telemetry flits and the local buffer capacity; and
a routing circuit configured to receive one or more packets via the one or more ingress ports, and route the one or more packets to the destination, via the plurality of egress ports, with bandwidth proportional to the network capacity.
2. The switch of claim 1 , wherein the local buffer capacity includes credit counts for the plurality of egress ports configured to couple the switch to a next switch.
3. The switch of claim 1 , wherein:
the telemetry circuit is configured to receive, from the plurality of next switches, link fault information for a plurality of links configured to couple the plurality of switches to one or more switches of the interconnection network; and
the network capacity circuit is configured to compute the network capacity further based on the link fault information.
4. The switch of claim 1 , wherein:
the telemetry tree includes a first set of next switches of the plurality of next switches, in a first level of the telemetry tree;
the telemetry tree includes a second set of next switches of the plurality of next switches, in a second level of the telemetry tree;
each switch in the first set of next switches is directly connected to the switch, in the topology of the interconnection network; and
each switch in the second set of next switches is indirectly connected to the switch, via the first set of next switches, in the topology of the interconnection network.
5. The switch of claim 4 , wherein the telemetry circuit is configured to generate the plurality of new telemetry flits by generating a telemetry block of flits that includes (i) per virtual lane buffer capacity information for each of the first set of next switches and (ii) consolidated buffer capacities for all virtual lanes for the second set of next switches.
6. The switch of claim 1 , wherein the telemetry circuit is configured to define bit-widths for measurements of buffer capacities in the plurality of new telemetry flits based on a telemetry update period.
7. The switch of claim 6 , wherein the telemetry circuit is configured to determine the telemetry update period based on buffer capacities of switches in the interconnect.
8. The switch of claim 1 , wherein the telemetry circuit is configured to define size of each of the plurality of new telemetry flits based on number of switches the switch is directly connected to.
9. The switch of claim 1 , wherein the telemetry circuit is configured to define size of each of the plurality of new telemetry flits based on predetermined congestion control bandwidth of the interconnection network.
10. The switch of claim 1 , wherein minimal or non-minimal values includes lane degrade.
11. The switch of claim 1 , wherein:
the plurality of telemetry flits includes cyclic redundancy check (CRC) information; and
the network capacity circuit is further configured to discard one or more telemetry flits, from the plurality of telemetry flits, according to the CRC information, while computing the network capacity.