Link fault isolation using latencies
Techniques are disclosed for identifying faulty links in a virtualized computing environment. Network path latency information is received for one or more network paths in the networked computing environment. Based on the network path latency information, a probable presence of a faulty component is determined. In response to the determination, physical links for a network path associated with the probable faulty component are identified. Information indicative of likely sources of the probable faulty component is received from multiple hosts of the networked computing environment. Based on the identified physical links and information, a faulty component is determined.
1. A method for identifying faulty links in a networked computing environment comprising a plurality of computing devices and network devices, the method comprising:
receiving round-trip latencies for network paths of the networked computing environment;
based on the round-trip latencies, determining a presence of a fault in the networked computing environment;
in response to the determination, executing traceroute functions to measure hop-to-hop latencies for physical links that are traversed on network paths of the networked computing environment;
receiving, from a plurality of hosts in the networked computing environment, votes that are indicative of which of the physical links are likely to be associated with the fault, wherein the votes are determined based on the hop-to-hop latencies; and
based on the votes, determining which of the physical links are associated with the fault.
2. The method of claim 1 , wherein the round-trip latencies are determined using RDMA transaction time stamps.
3. The method of claim 1 , wherein the determining which of the physical links are associated with the fault is further based on individual latency values associated with the physical links.
4. The method of claim 1 , wherein determining the presence of the fault is based on a value for an observed round-trip latency.
5. The method of claim 1 , further comprising scaling a number of votes associated with each physical link.
6. The method of claim 1 , wherein determining the presence of the fault comprises determining whether at least one of the round-trip latencies exceeds a threshold.
7. The method of claim 6 , wherein the determining whether the at least one of the round-trip latencies exceeds a threshold is based on a value for an observed round-trip latency.
8. The method of claim 7 , wherein the value is a fixed threshold.
9. The method of claim 7 , wherein the value is a minimum value.
10. The method of claim 7 , wherein the value is determined based on a statistically significant number of samples.
11. The method of claim 7 , wherein the value is determined dynamically per connection.
12. The method of claim 7 , wherein the value is adjusted based on a network queue size.
13. The method of claim 7 , wherein the value is determined based in part on congestion control data.
14. A system, comprising:
one or more processors; and
a memory in communication with the one or more processors, the memory having computer-readable instructions stored thereupon that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving round-trip latencies for network paths of the networked computing environment;
based on the round-trip latencies, determining a presence of a fault in the networked computing environment;
in response to the determination, executing traceroute functions to measure hop-to-hop latencies for physical links that are traversed on network paths of the networked computing environment;
receiving, from a plurality of hosts in the networked computing environment, votes that are indicative of which of the physical links are likely to be associated with the fault, wherein the votes are determined based on the hop-to-hop latencies; and
based on the votes, determining which of the physical links are associated with the fault.
15. The system of claim 14 , wherein the determining the presence of the fault comprises determining whether at least one of the round-trip latencies exceeds a threshold.
16. The system of claim 15 , wherein the threshold is one or more of a fixed threshold or a minimum value.
17. The system of claim 15 , wherein the threshold is determined based on a number of samples or congestion control data.
18. The system of claim 15 , wherein the threshold is determined dynamically per connection or adjusted based on a network queue size.
19. A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors of a computing device, cause the computing device to perform operations comprising:
receiving round-trip latencies for network paths of the networked computing environment;
based on the round-trip latencies, determining a presence of a fault in the networked computing environment;
in response to the determination, executing traceroute functions to measure hop-to-hop latencies for physical links that are traversed on network paths of the networked computing environment;
receiving, from a plurality of hosts in the networked computing environment, votes that are indicative of which of the physical links are likely to be associated with the fault, wherein the votes are determined based on the hop-to-hop latencies; and
based on the votes, determining which of the physical links are associated with the fault.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the determining which of the physical links are associated with the fault is based on an overlap of the network paths or based on associated latency values.