Autonomous distributed forwarding plane traceability based anomaly detection in application traffic for hyper-scale SD-WAN
Some embodiments of the invention provide a method for detecting and remediating anomalies in an SD-WAN that includes a controller, an enterprise datacenter, and multiple branch sites each having at least one edge node that includes a set of packet processing stages. At the controller, the method receives from a particular node of a particular branch site a flow notification indicating detection of an anomaly on the particular node. Based on the anomaly, the method dynamically generates trace monitoring rules that specify one or more flows to be traced and provides the trace monitoring rules to the particular node and at least one other node of another branch site. From the particular node and the at least one other node, the method receives trace monitoring results collected in response to the provided trace monitoring rules, and analyzes the results to identify any anomalies and dynamic actions to correct the anomalies.
1 . A method for tracing packets in a software-defined wide area network (SD-WAN) comprising a controller and a plurality of branch sites, wherein each branch site comprises at least a forwarding node comprising a set of packet processing stages that process packet flows that are forwarded through the SD-WAN, the method comprising:
at a particular forwarding node that comprises a plurality of packet processing stages:
receiving, from the controller of the SD-WAN, one or more trace monitoring rules specified for a particular packet flow, the one or more trace monitoring rules being dynamically generated based on a hierarchical order, the hierarchical order being associated with flow level and application level traceability;
determining that packets of the particular packet flow received at the particular forwarding node match at least one trace monitoring rule in the one or more trace monitoring rules;
based on the determination, directing each packet processing stage in the plurality of packet processing stages of the particular forwarding node to perform trace monitoring for the packets of the particular packet flow, each packet processing stage independently generating trace monitoring data, an anomaly detected at any packet processing stage triggering refining the one or more trace monitoring rules at the controller; and
forwarding a plurality of trace monitoring results for the particular packet flow to the controller for analysis.
2 . The method of claim 1 , wherein the plurality of packet processing stages comprises at least one forwarding operation and at least one middlebox service operation.
3 . The method of claim 1 , wherein the plurality of packet processing stages comprises at least two forwarding operations.
4 . The method of claim 1 , wherein the plurality of packet processing stages comprises at least two middlebox service operations.
5 . The method of claim 1 , further comprising setting a flag on each packet of the particular packet flow to indicate that the packet should be trace monitored.
6 . The method of claim 1 , wherein each packet processing stage in the plurality of packet processing stages generates trace monitoring data as the packets of the particular packet flow are processed.
7 . The method of claim 6 , wherein the set of one or more trace monitoring rules specifies a number of packets to be traced, wherein the plurality of trace monitoring results are forwarded to the controller when the specified number of packets have been traced.
8 . The method of claim 1 further comprising receiving one or more additional trace monitoring rules specified for one of (i) packet flows for a particular application, (ii) packet flows between a specified set of forwarding nodes in a plurality of forwarding nodes, and (iii) all packet flows between the plurality of forwarding nodes in the SD-WAN.
9 . The method of claim 1 , wherein the one or more trace monitoring rules further comprise a set of exception rules, wherein each exception rule defines a threshold value for identifying anomalous behavior on the particular forwarding node.
10 . The method of claim 1 further comprising:
receiving packet and determining that the packet does not belong to the particular flow; and
processing the packet without performing trace monitoring on the packet.
11 . A non-transitory machine readable medium storing a program for execution by a set of processing units, the program for tracing packets in a software-defined wide area network (SD-WAN) comprising a controller and a plurality of branch sites, wherein each branch site comprises at least a forwarding node comprising a set of packet processing stages that process packet flows that are forwarded through the SD-WAN, the program for execution at a particular forwarding node that comprises a plurality of packet processing stages and comprising sets of instructions for:
receiving, from the controller of the SD-WAN, one or more trace monitoring rules specified for a particular packet flow, the one or more trace monitoring rules being dynamically generated based on a hierarchical order, the hierarchical order being associated with flow level and application level traceability;
determining that the packets of the particular packet flow received at the particular forwarding node match at least one trace monitoring rule in the one or more trace monitoring rules;
based on the determination, directing each packet processing stage in the plurality of packet processing stages of the particular forwarding node to perform trace monitoring for the packets of the particular packet flow, each packet processing stage independently generating trace monitoring data, an anomaly detected at any packet processing stage triggering refining the one or more trace monitoring rules at the controller; and
forwarding a plurality of trace monitoring results for the particular packet flow to the controller for analysis.
12 . The non-transitory machine readable medium of claim 11 , wherein the plurality of packet processing stages comprises at least one forwarding operation and at least one middlebox service operation.
13 . The non-transitory machine readable medium of claim 11 , wherein the plurality of packet processing stages comprises at least two forwarding operations.
14 . The non-transitory machine readable medium of claim 11 , wherein the plurality of packet processing stages comprises at least two middlebox service operations.
15 . The non-transitory machine readable medium of claim 11 , further comprising a set of instructions for setting a flag on each packet of the particular packet flow to indicate that the packet should be trace monitored.
16 . The non-transitory machine readable medium of claim 11 , wherein each packet processing stage in the plurality of packet processing stages generates trace monitoring results data as the packets of the particular packet flow are processed.
17 . The non-transitory machine readable medium of claim 16 , wherein the one or more trace monitoring rules specifies a number of packets to be traced, wherein the plurality of trace monitoring results are forwarded to the controller when the specified number of packets have been traced.
18 . The non-transitory machine readable medium of claim 11 , wherein the controller (i) analyzes the forwarded plurality of trace monitoring results and (ii) performs a dynamic action when an anomaly is identified, wherein the dynamic action auto-corrects the identified anomaly.
19 . The non-transitory machine readable medium of claim 11 , wherein the program further comprises a set of instructions for receiving one or more additional trace monitoring rules specified for one of (i) packet flows for a particular application, (ii) packet flows between a specified set of forwarding nodes in a plurality of forwarding nodes, and (iii) all packet flows between the plurality of forwarding nodes in the SD-WAN.
20 . A forwarding node to trace packets in a software-defined wide area network (SD-WAN) comprising a controller and branch sites which include other forwarding nodes having sets of packet processing stages that process packet flows that are forwarded through the SD-WAN, the forwarding node comprising a set of packet processing stages constructed and arranged to:
receive, from the controller of the SD-WAN, one or more trace monitoring rules specified for a particular packet flow, the one or more trace monitoring rules being dynamically generated based on a hierarchical order, the hierarchical order being associated with flow level and application level traceability;
determine that the packets of the particular packet flow received at the forwarding node match at least one trace monitoring rule in the one or more trace monitoring rules;
based on the determination, direct the packet processing stages of the forwarding node to perform trace monitoring for the packets of the particular packet flow, each packet processing stage independently generating trace monitoring data, an anomaly detected at any packet processing stage triggering refining the one or more trace monitoring rules at the controller; and
forwarding a plurality of trace monitoring results for the particular packet flow to the controller for analysis.