Managing anomalous network telemetry message flows
View Patent ↗A process includes monitoring a plurality of message flows sent by respective network devices of a plurality of network devices. The plurality of message flows is associated with reporting a network telemetry metric to a network management system. The process includes determining that a given message flow of the plurality of message flows exhibits an unexpected behavior. The process includes, responsive to determining that the given message flow exhibits the unexpected behavior, determining an aggregate available bandwidth for message flows of the plurality of message flows, which respectively exhibit expected behaviors. The process includes, responsive to determining that the given message flow exhibits the unexpected behavior, modifying a bandwidth of the given message flow based on the aggregate available bandwidth.
1 . A method comprising:
monitoring, by a message rate tracking engine, a plurality of message flows sent by respective network devices of a plurality of network devices, wherein the plurality of message flows is associated with reporting a network telemetry metric to a network management system;
determining, by the message rate tracking engine, that a given message flow of the plurality of message flows exhibits an unexpected behavior; and
responsive to determining that the given message flow exhibits the unexpected behavior:
determining, by a subscription management engine, an aggregate available bandwidth for message flows of the plurality of message flows, which respectively exhibit expected behaviors;
determining, by the subscription management engine, an adjusted bandwidth for the given message flow based on the aggregate available bandwidth; and
modifying, by the subscription management engine, a subscription of the given network device to the network management system based on the adjusted bandwidth.
2 . The method of claim 1 , wherein determining that the given message flow exhibits the unexpected behavior comprises comparing a message rate of the given message flow with a message rate threshold.
3 . The method of claim 2 , further comprising determining message rate threshold, wherein determining the message rate threshold comprises applying a baselining algorithm to a plurality of message rates associated with the network telemetry metric and associated with a plurality of network devices of a local network that contains the given network device.
4 . The method of claim 2 , further comprising determining message rate threshold, wherein determining the message rate threshold comprises applying a baselining algorithm to a plurality of message rates associated with the network telemetry metric and associated with a plurality of network devices located in at least one other local network than a local network that contains the given network device.
5 . The method of claim 2 , wherein a second network device of the plurality of network devices sends messages of a network telemetry message flow pursuant to a periodic subscription interval, the method further comprising:
responsive to determining that the given message flow exhibits the unexpected behavior, modifying a subscription of a second network device of the plurality of network devices to increase a periodic subscription interval at which the second network device sends messages of the respective message flow.
6 . The method of claim 1 , further comprising:
responsive to determining that the given message flow exhibits the unexpected behavior, modifying, by the subscription manager, a second subscription of a periodic message flow associated with reporting a second network telemetry metric to the network management system, wherein modifying the second subscription comprises increasing a period of the periodic subscription from a first reporting time interval to a second time reporting interval to decrease a bandwidth of the periodic message flow.
7 . The method of claim 6 , further comprising:
receiving, by the network management system, the periodic message flow corresponding to the second reporting time interval to provide a received periodic message flow; and
imputing, by the network management system, content missing in the received periodic message flow corresponding to the difference between the first reporting time interval and the second reporting time interval.
8 . The method of claim 6 , wherein:
the received periodic message flow comprises first samples of a network telemetry statistic;
the imputing comprises estimating additional second samples of the network telemetry statistic corresponding to the difference between the first reporting time interval and the second reporting time interval.
9 . The method of claim 1 , further comprising:
receiving, by the network management system, the given message flow having the adjusted bandwidth to provide a received given message flow; and
imputing, by the network management system, content missing in the received given message flow corresponding to the difference between a bandwidth of the given message flow before the adjustment of the bandwidth and the adjusted bandwidth.
10 . The method of claim 1 , wherein the subscription before modification allows the given network device to report the network telemetry metric responsive to a state of the network telemetry metric changing, and modifying the subscription comprises constraining reporting of the network telemetry metric by the network device to a time window having a duration that corresponds to the adjusted bandwidth.
11 . The method of claim 10 , further comprising responsive to expiration of the time window:
determining whether the given message flow continues to exhibit an unexpected behavior; and
responsive to determining that the given message flow continues to exhibit the unexpected behavior, constraining reporting of the network telemetry metric by the network device to another time window.