Fabric control protocol for large-scale multi-stage data center networks
A fabric control protocol (FCP) and packet forwarding mechanisms are described that maximize utilization of bandwidth within massive, large-scale data centers having multi-stage data center switch fabric topologies, such as topologies that include a third switching layer formed by super spine switches. Automatic generation of data plane forwarding information referred to as FCP path information enumerates, for each data processing unit (DPU), the available FCP paths. The FCP path information may be based on unique combinations of peak points of the switch fabric for a given DPU with FCP colors assigned to network links that are used to multi-home the DPU to the switch fabric.
1. A data center network comprising:
a plurality of servers;
a multi-stage data center switch fabric comprising a plurality of switches arranged in at least three tiers;
a plurality of data processing units (DPUs), each of the DPUs connecting a subset of one or more of the servers to the data center switch fabric, wherein the DPUs include a source DPU connected to a source server of the plurality of servers and a destination DPU coupled to a destination server of the plurality of servers, wherein the source DPU and the destination DPU are each configured to execute a fabric control protocol (FCP) to establish an FCP tunnel associated with a plurality of parallel paths across the data center switch fabric between the source DPU and the destination DPU;
wherein, for each packet of a packet flow of packets from the source server to the destination server, the source DPU is configured to:
(i) select, based on FCP path information, a respective one of the plurality of parallel paths for each of the packets, wherein the FCP path information identifies each of the plurality of parallel paths for the FCP tunnel between the source DPU and the destination DPU,
(ii) construct FCP packets to encapsulate each of the packets within payloads of the FCP packets, and
(iii) spray the FCP packets over the plurality of parallel paths of the FCP tunnel by forwarding each the FCP packets through the switch fabric across the respective one of the plurality of parallel paths selected for the packet.
2. The data center network of claim 1 , wherein the data center switch fabric comprises at least three tiers including:
a set of top-of-rack (ToR) switches forming a first tier of the switch fabric, each of the ToR switches coupled to one or more of the DPUs,
a set of spine switches forming a second tier, each of the spine switches coupled to one or more ToR switches, and
a set of super spine switches forming a third tier, each of the super spine switches interconnecting two or more of the spine switches.
3. The data center network of claim 1 ,
wherein the FCP path information comprises an enumeration of identifiers that is common to each of the DPUs,
wherein, for each DPU, the identifiers of the FCP path information identify a set of FCP legs that connect the respective DPU to peak points in the multi-stage data center switch fabric,
wherein each of the peak points represents one of the switching devices of the data center that is a midway point along a path through the multi-stage switch fabric between the respective DPU and a second one of the DPUs.
4. The data center network of claim 3 ,
wherein each DPU is multi-homed such that the each DPU is connected to the switch fabric by a plurality of network links, and
wherein, for each DPU, the identifiers of the FCP path information identify the FCP legs as unique combinations of (i) a peak point selected from the set of peak points within the switch fabric, and (ii) an FCP color selected from a set of FCP colors, each FCP color representing a different one of the network links coupling the DPU to the switch fabric.
5. The data center network of claim 4 ,
wherein the data center switch fabric comprises at least three tiers, each of the tiers having a plurality of switches comprising (i) a first tier of leaf switches and a second tier of spine switches arranged in a plurality of modular units of physical infrastructure (“PoDs”), and (ii) a third tier of super spine switches coupled to the spine switches of the PoDs arranged in a set of switch planes,
wherein, for each of the PoDs, the spine switches for the POD are connected to super spine switches in different switch planes, and
wherein, for each DPU, the FCP path information comprises T identifiers, wherein the T identifiers identify the FCP legs as T unique combinations of (i) M peak points within the switch fabric for the DPU, and (ii) an FCP color selected from a set of N FCP colors, each FCP color representing a different one of the N network links, such that T=M*N.
6. The data center network of claim 4 ,
wherein when selecting a respective one of the plurality of parallel paths for each of the packets, the source DPU is configured to select one of the identifiers specified in the FCP path information, the identifier identifying one of the peak points and one of the FCP colors,
wherein when constructing each of the FCP packets, the source DPU is configured to construct each of the FCP packets to include a deterministic forwarding label that represents the peak point and the FCP color represented by the FCP path information identifier selected for the packet, and
wherein when spraying the FCP packets, the source DPU forwards each of the packets using the network link assigned the FCP color for the FCP path information identifier selected for the packet.
7. The data center network of claim 4 ,
wherein when selecting a respective one of the plurality of parallel paths for each of the packets, the source DPU is configured to select one of the identifiers specified in the FCP path information, the identifier identifying one of the peak points and one of the FCP colors,
wherein when constructing each of the FCP packets, the source DPU is configured to construct the FCP packet to include an inner FCP label and an outer FCP label selected according to the FCP path information identifier selected for the packet,
wherein the inner FCP label specifies a destination network address for the network interface of the destination DPU that has been assigned the FCP color according to the FCP path information identifier selected for the packet, and
wherein the outer FCP label specifying a destination network address of the switching device within the data center that corresponds to the peak points according to the FCP path information identifier selected for the packet.
8. The data center network of claim 1 , further comprising a network controller configured to automatically generate the FCP path information for each of the DPUs of the data center network.
9. The data center network of claim 1 ,
wherein the source DPU is configured to send an FCP request message for an amount of data to be transferred, and
wherein the destination DPU is configured to, in response to receipt of an FCP request message, perform grant scheduling and send an FCP grant message indicating the amount of bandwidth reserved for the source DPU for the FCP tunnel.
10. The data center network of claim 1 ,
wherein, when operating in an unsolicited mode, the source DPU is configured to spray the FCP packets across the plurality of parallel paths of the FCP tunnel without sending a prior FCP request message,
wherein the destination DPU is configured to, in response to receipt of the FCP packets without receiving the prior FCP request message, send an FCP grant message to the sender node including an acknowledgment (ACK) for a last received FCP packet of the packet flow, and
wherein, in response to receipt of the rFCP grant message, the sender node modifies a send window size for the FCP tunnel.
11. A method comprising:
executing a fabric control protocol (FCP) to establish a logical tunnel over a plurality of parallel paths between a source DPU and a destination DPU in a multi-stage data center switch fabric comprising a plurality of switches arranged in at least three tiers, wherein the source DPU is coupled to a source server and the destination DPU is coupled to a destination server,
receiving, with the source DPU, a packet flow of packets from the source server to be forwarded through the data center switch fabric to the destination server coupled to the destination DPU;
selecting, based on FCP path information, a respective one of the plurality of parallel paths for each of the packets, wherein the FCP path information identifies each of the plurality of parallel paths for the FCP tunnel between the source DPU and the destination DPU;
constructing FCP packets to encapsulate each of the packets within payloads of FCP packets; and
spraying the FCP packets over the plurality of parallel paths of the FCP tunnel by forwarding each the FCP packets through the switch fabric across the respective one of the plurality of parallel paths selected for the packet.
12. The method of claim 11 , wherein the data center switch fabric comprises at least three tiers including:
a set of top-of-rack (ToR) switches forming a first tier of the switch fabric, each of the ToR switches coupled to one or more of the DPUs,
a set of spine switches forming a second tier, each of the spine switches coupled to one or more ToR switches, and
a set of super spine switches forming a third tier, each of the super spine switches interconnecting two or more of the spine switches.
13. The method of claim 11 ,
wherein the FCP path information comprises an enumeration of identifiers that is common to each of the DPUs,
wherein, for each DPU, the identifiers of the FCP path information identify a set of FCP legs that connect the respective DPU to peak points in the multi-stage data center switch fabric,
wherein each of the peak points represents one of the switching devices of the data center that is a midway point along a path through the multi-stage switch fabric between the respective DPU and a second one of the DPUs.
14. The method of claim 13 ,
wherein each DPU is multi-homed such that the each DPU is connected to the switch fabric by a plurality of network links, and
wherein, for each DPU, the identifiers of the FCP path information identify the FCP legs as unique combinations of (i) a peak point selected from the set of peak points within the switch fabric, and (ii) an FCP color selected from a set of FCP colors, each FCP color representing a different one of the network links coupling the DPU to the switch fabric.
15. The method of claim 14 ,
wherein the data center switch fabric comprises at least three tiers, each of the tiers having a plurality of switches comprising (i) a first tier of leaf switches and a second tier of spine switches arranged in plurality of modular units of physical infrastructure (“PoDs”), and (ii) a third tier of super spine switches coupled to the spine switches of the PoDs arranged in a set of switch planes,
wherein, for each of the PoDs, the spine switches for the POD are connected to super spine switches in different switch planes, and
wherein, for each DPU, the FCP path information comprises T identifiers, wherein the T identifiers identify the FCP legs as T unique combinations of (i) M peak points within the switch fabric for the DPU, and (ii) an FCP color selected from a set of N FCP colors, each FCP color representing a different one of the N network links, such that T=M*N.
16. The method of claim 15 ,
wherein selecting a respective one of the plurality of parallel paths for each of the packets comprises selecting, with the source DPU, one of the identifiers specified in the FCP path information, the identifier identifying one of the peak points and one of the FCP colors,
wherein constructing each of the FCP packets comprises constructing, with the source DPU, each of the FCP packets to include an deterministic forwarding label that represents the peak point and the FCP color represented by the FCP path information identifier selected for the packet, and
wherein spraying the FCP packets comprises forwarding, with the source DPU, each of the packets on the network link assigned the FCP color for the FCP path information identifier selected for the packet.
17. The method of claim 14 ,
wherein selecting a respective one of the plurality of parallel paths for each of the packets comprises selecting, with the source DPU, one of the identifiers specified in the FCP path information, the identifier identifying one of the peak points and one of the FCP colors, and
wherein constructing each of the FCP packets comprises constructing, with the source DPU, the FCP packet to include an inner FCP label and an outer FCP label selected according to the FCP path information identifier selected for the packet, the inner FCP label specifying a destination network address for the network interface of the destination DPU coupled to the link that has been assigned the FCP color specified by the FCP path information identifier selected for the FCP packet, and the outer FCP label specifying a destination network address of the switching device within the data center that corresponds to the peak points according to the FCP path information identifier selected for the packet.
18. The method of claim 14 , further comprising, when constructing the FCP packets, dynamically computing the destination network address for the network interface of the destination DPU based on the FCP Color specified for the selected identifier of the FCP path information.
19. The method of claim 11 , further comprising automatically generating the FCP path information for each of the DPUs of the data center network with a network controller.
20. A method of generating fabric control protocol (FCP) path information for a data center having a plurality of nodes, a multi-stage data center switch fabric having a plurality of switches arranged in at least three tiers, and a plurality of data processing units (DPUs), each of the DPUs multi-homed to the data center switch fabric by a plurality of links and connecting a subset of one or more of the nodes to the data center switch fabric, the method comprising:
determining, for each pair of DPUs in the data center, a set of peak points within the data center switch fabric, each of the peak points representing one of the switching devices of the data center that is a midway point along a path through the multi-stage switch fabric between the respective DPU and a second one of the DPUs;
determining, for each DPU, a set of FCP legs within the data center switch fabric, each FCP leg representing a unique path that connects the DPU to one peak point of the set of peak points in the multi-stage data center switch fabric;
determining a set of FCP colors representing the number of links used to multi-home the DPUs;
assigning, for each DPU, a different one of the FCP colors to each of the links coupling the DPU to the switch fabric;
generating, for each DPU, FCP path information having a set of identifiers, each identifier representing unique combinations of (i) a peak point selected from the set of peak points within the switch fabric for the DPU, and (ii) the FCP colors assigned to the different links coupling the DPU to the switch fabric; and
installing the corresponding FCP path information within the respective DPU for which the FCP path information was generated to control packet forwarding by the DPU.
21. The method of claim 20 , wherein each node comprises one or more of a server, a storage node, a compute node, a virtual machine or a container.
22. The method of claim 20 , wherein the method is automatically performed by a network controller.
23. The method of claim 20 , wherein the data center switch fabric comprises at least three tiers including:
a set of top-of-rack (ToR) switches forming a first tier of the switch fabric, each of the ToR switches coupled to one or more of the DPUs,
a set of spine switches forming a second tier, each of the spine switches coupled to one or more ToR switches, and
a set of super spine switches forming a third tier, each of the super spine switches interconnecting two or more of the spine switches.