Automated datacenter network failure mitigation
The subject disclosure is directed towards a technology that automatically mitigates datacenter failures, instead of relying on human intervention to diagnose and repair the network. Via a mitigation pipeline, when a network failure is detected, a candidate set of components that are likely to be the cause of the failure is identified, with mitigation actions iteratively targeting each component to attempt to alleviate the problem. The impact to the network is estimated to ensure that the redundancy present in the network will be able to handle the mitigation action without adverse disruption to the network.
1. A method performed at least in part by at least one processor, comprising:
monitoring a network;
determining a network component set corresponding to abnormal behavior, in which the network component set comprises a plurality of links; and
taking automated action on the network component set to mitigate the abnormal behavior by:
identifying a plurality of proposed actions;
estimating an impact on the network for executing each of the proposed actions;
estimating a success rate for each of the proposed actions selecting one of more of the proposed actions based at least on the estimated impact and the estimated success rate; and
applying the one or more of the proposed actions based at least on the estimated impact and the estimated success rate.
2. The method of claim 1 , further comprising:
selecting two or more of the proposed actions based at least on the estimated impact and the estimated success rate.
3. The method of claim 2 , wherein estimating the impact comprises estimating a result of restarting or deactivating a component of the network.
4. The method of claim 3 , wherein estimating the impact comprises using at least some historical data.
5. The method of claim 1 , wherein the abnormal behavior is a link failure that causes traffic latency, traffic overload, or traffic imbalance between a first link and a second link in the plurality of links.
6. The method of claim 5 , further comprising, restarting the first link or the second link as a result of determining the abnormal behavior causes traffic imbalance.
7. The method of claim 5 , further comprising restarting a switch corresponding to the first link or the second link as a result of determining the abnormal behavior.
8. The method of claim 1 , further comprising providing a plan comprising at least one action, and wherein taking the action comprises executing the plan.
9. The method of claim 1 , wherein taking the action comprises restarting or deactivating at least one component of the network component set.
10. The method of claim 1 , wherein the abnormal behavior corresponds to a link-down failure or uneven-split failure, and wherein providing the plan comprises deactivating or restarting one of the plurality of links.
11. A system comprising:
a detector configured to process network state data for a network to determine a state indicative of abnormal behavior in a network component set comprising a plurality of links;
a planner configured to determine a plan for mitigating the abnormal behavior by identifying a plurality of proposed actions, the planner coupled to an impact estimator configured to:
estimate an impact on the network for executing each of the proposed actions; and
estimate a success rate for each of the proposed actions; and
a plan executor, the plan executor configured to access the plan and take one or more actions identified in the plan on one of the plurality of links to mitigate the abnormal behavior by applying one or more of the proposed actions based at least on the estimated impact and the estimated success rate.
12. The system of claim 11 , wherein the detector is coupled to an aggregator, the aggregator configured to decide whether a detected abnormal behavior corresponds to a new or an ongoing abnormal behavior.
13. The system of claim 11 , further comprising a database, the planner coupled to the database to obtain data corresponding to the abnormal behavior and to output the plan to the database.
14. The system of claim 11 , wherein the plan includes a command to restart a link.
15. The system of claim 11 , wherein the component set further includes a switch, and wherein the plan further includes a command to restart or deactivate the switch.
16. The system of claim 11 , wherein the detector monitors SNMP data and one or more performance counters, or both, to detect abnormal behavior comprising: a, and traffic latency or traffic imbalance.
17. One or more computer-readable devices having computer-executable instructions, which when executed by at least one computer perform operations comprising:
monitoring a network;
determining a network component set corresponding to abnormal behavior, in which the network component set comprises a plurality of links; and
taking automated action on the network component set to mitigate the abnormal behavior by:
identifying a plan comprising a plurality of proposed actions;
estimating an impact on the network for executing each of the proposed actions;
estimating a success rate for each of the proposed actions; and
applying one or more of the proposed actions based at least on the estimated impact and the estimated success rate.
18. The one or more computer-readable devices of claim 17 having further computer-executable instructions comprising, determining whether the abnormal behavior corresponds to a new or an ongoing abnormal behavior.
19. The one or more computer-readable devices of claim 17 , wherein the plan comprises deactivating or restarting one of the plurality of links.
20. The one or more computer-readable devices of claim 17 , wherein the plan comprises deactivating or restarting a switch.