Resilient network communication using selective multipath packet flow spraying
Techniques for detecting path failures and reducing packet loss as a result of such failures are described for use within a data center or other environment. For example, a source and/or destination access node may create and/or maintain information about health and/or connectivity for a plurality of ports or paths between the source and destination device and core switches. The source access node may spray packets over a number of paths between the source access node and the destination access node. The source access node may use the information about connectivity for the paths between the source or destination access nodes and the core switches to limit the paths over which packets are sprayed. The source access node may spray packets over paths between the source access node and the destination access node that are identified as healthy, while avoiding paths that have been identified as failed.
1. A network system comprising:
a plurality of servers, including a source server and a destination server;
a switch fabric comprising a plurality of core switches; and
a plurality of access nodes configured to forward packets between the servers, wherein each of the access nodes is coupled to a subset of the servers and to a subset of the core switches, and wherein the plurality of access nodes includes a source access node coupled to the source server and a destination access node coupled to the destination server,
wherein, when communicating a packet flow of packets between the source server and the destination server, the source access node is configured to spray the packets of the packet flow across a plurality of data paths to the destination access node, and
wherein the destination access node is configured to:
identify information about failures of one more of a plurality of destination paths, wherein each of the destination paths is a path between the destination access node and one of the core switches,
receive a data transfer request message originating at the source access node, wherein the data transfer request message identifies an amount of data that the source access node has requested to transfer, and
responsive to receiving the request message, send a data transfer grant message destined for the source access node, wherein the grant message includes a vector specifying port health information for each of the plurality of destination paths, wherein the vector identifies which of the plurality of destination paths are inoperable.
2. The network system of claim 1 , wherein the source access node is further configured to:
receive the data transfer grant message;
identify, based on the vector specifying port health information included within the data transfer grant message, a set of failed data paths to the destination access node, and
wherein to spray the packets of the packet flow, the source access node sprays the packets over a subset of all data paths to the destination access node, wherein the subset of all data paths excludes the set of failed data paths.
3. The network system of claim 1 , wherein the source access node is further configured to:
identify information about source path failures between the source access node and one of the core switches;
receive the data transfer grant message;
identify, based on the vector specifying port health information included within the data transfer grant message and further based on the information about source path failures, a set of failed data paths to the destination access node; and
wherein to spray the packets of the packet flow, the source access node sprays packets over a subset of all data paths to the destination access node, wherein the subset of all data paths excludes the set of failed data paths.
4. The network system of claim 3 , wherein the source access node is further configured to:
determine a request rate limiter based on how many data paths from the source access node are not identified as failed data paths, and further based on a count of all data paths from the source access node.
5. The network system of claim 3 , wherein the destination access node is further configured to:
determine a grant data rate limiter based on how many data paths from the destination access node are not identified as failed data paths, and further based on a count of all data paths from the destination access node.
6. The network system of claim 3 , wherein the data transfer request message includes information about a request rate, and wherein the destination access node is further configured to:
reserve, based on the information about the request rate and further based on the vector specifying port health information, bandwidth for a data transfer from the source device.
7. The network system of claim 6 , wherein the information about the request rate is based on information about source path failures.
8. The network system of claim 1 , wherein the vector specifying port health information includes a multi-dimensional array having a first dimension corresponding to each of the plurality of access nodes, and a second dimension corresponding to network interface ports on each of the plurality of access nodes.
9. The network system of claim 1 , wherein the destination path that are inoperable failures include at least one of:
a failure of at least one of the plurality access nodes, or
a failure of a link between at least one of the access nodes and at least one of the core switches.
10. The network system of claim 1 , wherein the source access node is configured to:
identify information about source path failures between the source access node and the core switches; and
perform, based on the information about source path failures, diagnostic testing.
11. A method comprising:
spraying, by a source access node coupled to a source server, packets of a packet flow across a plurality of data paths to a destination access node coupled to a destination server, wherein the source access node and the destination access node are each included within a plurality of access nodes configured to forward packets between the source server and the destination server, and wherein each of the plurality of access nodes is coupled to a subset of a plurality of core switches;
identifying, by the destination access node, information about failures of one or more of a plurality of destination paths, wherein each of the destination paths is a path between the destination access node one of the plurality of core switches;
receiving, by the destination access node, a data transfer request message originating at the source access node, wherein the data transfer request message identifies an amount of data that the source access node has requested to transfer; and
responsive to receiving the request message, sending, by the destination access node, a data transfer grant message destined for the source access node, wherein the grant message includes a vector specifying port health information for each of the plurality of destination paths, wherein the vector identifies which of the plurality of destination paths are inoperable.
12. The method of claim 11 , further comprising:
receiving, by the source access node, the data transfer grant message; and
identifying, by the source access node and based on the vector specifying port health information included within the data transfer grant message, a set of failed data paths to the destination access node, and
wherein spraying the packets of the packet flow includes spraying the packets over a subset of all data paths to the destination access node, wherein the subset of all data paths excludes the set of failed data paths.
13. The method of claim 11 , further comprising:
identifying, by the source access node, information about source path failures between the source access node and the core switches;
receiving, by the source access node, the data transfer grant message; and
identifying, by the source access node and based on the vector specifying port health information included within the data transfer grant message and further based on the information about source path failures, a set of failed data paths to the destination access node, and
wherein spraying the packets of the packet flow includes spraying packets over a subset of all data paths to the destination access node, wherein the subset of all data paths excludes the set of failed data paths.
14. The method of claim 13 , further comprising:
determining, by the source access node, a request rate limiter based on how many data paths from the source access node are not identified as failed data paths, and further based on a count of all data paths from the source access node.
15. The method of claim 13 , further comprising:
determining, by the destination access node, a grant data rate limiter based on how many data paths from the destination access node are not identified as failed data paths, and further based on a count of all data paths from the destination access node.
16. The method of claim 13 , wherein the data transfer request message includes information about a request rate, the method further comprising:
reserving, by the destination access node and based on the information about the request rate and further based on the vector specifying port health information, bandwidth for a data transfer from the source device.
17. The method of claim 16 , wherein the information about the request rate is based on information about source path failures.
18. A system comprising:
a source access node coupled to a source server and coupled to a first plurality of core switches; and
a destination access node coupled to a destination server and coupled to a second plurality of core switches, wherein the second plurality of core switches includes at least one of the core switches in the first plurality of core switches, and wherein the destination access node is configured to:
identify information about failures of one or more of a plurality of destination paths, wherein each of the destination paths is a path between the destination access node and one of the second plurality of core switches,
receive a data transfer request message originating at the source access node, wherein the data transfer request message includes an amount of data that the source access node has requested to transfer, and
responsive to receiving the request message, send a data transfer grant message destined for the source access node, wherein the grant message identifies a vector specifying port health information for each of the plurality of destination paths, wherein the vector identifies which of the plurality of destination paths are inoperable.
19. The system of claim 18 , wherein the source access node is configured to:
receive the data transfer grant message;
identify, based on the vector specifying port health information included within the data transfer grant message, a set of failed data paths to the destination access node; and
spray packets over a subset of all data paths to the destination access node, wherein the subset of all data paths excludes the set of failed data paths.
20. The system of claim 18 , wherein the source access node is configured to:
identify information about source path failures between the source access node and the core switches;
receive the data transfer grant message;
identify, based on the vector specifying port health information included within the data transfer grant message and further based on the information about source path failures, a set of failed data paths to the destination access node; and
spray packets over a subset of all data paths to the destination access node, wherein the subset of all data paths excludes the set of failed data paths.