Link fault isolation using latencies
Techniques are disclosed for identifying faulty links in a virtualized computing environment. Network path latency information is received for one or more network paths in the networked computing environment. Based on the network path latency information, a probable presence of a faulty component is determined. In response to the determination, physical links for a network path associated with the probable faulty component are identified. Information indicative of likely sources of the probable faulty component is received from multiple hosts of the networked computing environment. Based on the identified physical links and information, a faulty component is determined.
1. A system, comprising:
one or more processors; and
a memory in communication with the one or more processors, the memory having computer-readable instructions stored thereupon that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving round-trip latencies to a destination over one or more network paths communicatively coupled to the system;
determining whether at least one of the round-trip latencies exceeds a threshold comprising a fixed or variable value;
in response to determining that at least one of the round-trip latencies exceeds the threshold, executing a traceroute function to measure hop-to-hop latencies for physical links on network paths where the round-trip latencies exceed the threshold for connections to the destination;
based on the hop-to-hop latencies, identifying one of the physical links that has a highest observed hop-to-hop latency; and
sending, to a central collecting function, information indicative of the identified physical link, wherein the central collecting function is configured to receive the information indicative of the identified physical link and determine a faulty link of the physical links using a voting mechanism and based on the information indicative of the identified physical link and additional information for physical links identified by other nodes communicatively coupled to the network paths.
2. The system of claim 1 , wherein the determining whether at least one of the round-trip latencies exceeds a threshold is based on an observed round-trip latency.
3. The system of claim 1 , wherein the threshold is a fixed threshold.
4. The system of claim 1 , wherein the threshold is a minimum value.
5. The system of claim 1 , wherein the threshold is determined based on a number of samples.
6. The system of claim 1 , wherein the threshold is determined dynamically per connection.
7. The system of claim 1 , wherein the threshold is adjusted based on a network queue size.
8. The system of claim 1 , wherein the threshold is determined based in part on congestion control data.
9. A method for identifying faulty links in a networked computing environment comprising one or more computing devices and one or more network devices, the method comprising:
receiving round-trip latencies to a destination over one or more network paths communicatively coupled to the one or more computing devices;
determining whether at least one of the round-trip latencies exceeds a threshold or exceeds characteristic latencies for connections to the same destination;
in response to determining that at least one of the round-trip latencies exceeds the threshold or characteristic latencies, executing a traceroute function to measure hop-to-hop latencies for physical links on network paths where the round-trip latencies exceed the threshold or exceed the characteristic latencies for connections to the destination;
based on the hop-to-hop latencies, identifying one of the physical links that has a highest observed hop-to-hop latency; and
sending, to a central collecting function, information indicative of the identified physical link, wherein the central collecting function is configured to receive the information indicative of the identified physical link and determine a faulty link of the physical links using a voting mechanism and based on the information indicative of the identified physical link and additional information for physical links identified by other nodes communicatively coupled to the network paths.
10. The method of claim 9 , wherein the round-trip latencies are determined using RDMA transaction time stamps.
11. The method of claim 9 , wherein the determination of the faulty link is based on individual latency values associated with the physical links.
12. The method of claim 9 , wherein the determination of the faulty link is based on a value for an observed round-trip latency.
13. The method of claim 9 , wherein the determining whether at least one of the round-trip latencies exceeds a threshold or exceeds characteristic latencies for connections to the same destination is based on a value for an observed round-trip latency.
14. A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors of a computing device, cause the computing device to perform operations comprising:
receiving round-trip latencies to a destination over one or more network paths communicatively coupled to a networked computing environment comprising one or more computing devices and one or more network devices;
determining whether at least one of the round-trip latencies exceeds a threshold or exceeds characteristic latencies for connections to the same destination;
in response to determining that at least one of the round-trip latencies exceeds the threshold or characteristic latencies, executing a traceroute function to measure hop-to-hop latencies for physical links on network paths where the round-trip latencies exceeds the threshold or exceeds the characteristic latencies for connections to the destination;
based on the hop-to-hop latencies, identifying one of the physical links that has a highest observed hop-to-hop latency; and
sending, to a central collecting function, information indicative of the identified physical link, wherein the central collecting function is configured to receive the information indicative of the identified physical link and determine a faulty link of the physical links using a voting mechanism and based on the information indicative of the identified physical link and additional information for physical links identified by other nodes communicatively coupled to the network paths.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the determination of the faulty link is determined using RDMA transaction time stamps.
16. The non-transitory computer-readable storage medium of claim 14 , wherein the determination of the faulty link is based on an overlap of the network paths.
17. The non-transitory computer-readable storage medium of claim 14 , wherein the determination of the faulty link is based on associated latency values.
18. The non-transitory computer-readable storage medium of claim 14 , wherein the threshold is a fixed threshold or a minimum value.
19. The non-transitory computer-readable storage medium of claim 14 , wherein the threshold is determined based on a number of samples, determined dynamically per connection, based on a network queue size, or determined based in part on congestion control data.
20. The non-transitory computer-readable storage medium of claim 14 , wherein the determining whether at least one of the round-trip latencies exceeds a threshold or exceeds characteristic latencies for connections to the same destination is based on a value for an observed round-trip latency.