Method and system for confident anomaly detection in computer network traffic
The present invention relates to systems and methods for detecting anomalies in computer network traffic with fewer false positives and without the need for time-consuming and unreliable historical baselines. Upon detection, traffic anomalies can be processed to determine valuable network insights, including health of interfaces, devices and network services, as well as to provide timely alerts in the event of attack.
1. A method of processing network metadata comprising the steps of:
receiving network metadata from a plurality of sources in a data processing system, in at least one data format;
determining the network metadata type;
converting at least a portion of said network metadata into at least one different data formats that are used in the data processing system for other system metadata based upon the type;
processing the network metadata while the network metadata is in transition on a network between a network device that generated the network metadata and a device that is able to store the network metadata, wherein the processing includes:
receiving a plurality of flow records each containing IP addresses of communicating endpoints and an IP address of a flow reporting device of a plurality of flow reporting devices which provided a flow record;
noting the time when each flow record was received;
storing information about the IP addresses of communicating endpoints, IP addresses of the plurality of flow reporting devices and the time of last observed communication as an indicator of the frequency of at least some communication paths utilized by the plurality of flow reporting devices;
designating, based upon the stored information of a flow reporting device with the highest frequency of use among the plurality of flow reporting devices, the highest frequency flow reporting device as an authoritative source of information about communications between the communicating endpoints;
forwarding for further processing a flow record about communication between the communicating endpoints received from the authoritative source, wherein the further processing includes computation of a network health score as a fuzzy classifier of an inference matrix of relative packet rate values and relative traffic volume for the authoritative source;
emitting an alert if the result of the further processing signifies a network traffic anomaly; and
discarding flow records about communication between the communicating endpoints received from other flow reporting devices.
2. The method of claim 1 , wherein the relative packet rate values and the relative traffic volume are classified as low, medium or high respectively.
3. The method of claim 1 , further comprising generating an attention level based upon the inference matrix.
4. The method of claim 3 , wherein the network health score is calculated as one minus the attention level.
5. The method of claim 3 , wherein the attention level is calculated using Mamdani and Sugeno fuzzy inference methods.
6. The method of claim 5 , wherein the attention level is calculated as an average attention level of a Mamdani attention level calculation and a Sugeno attention level calculation.
7. The method of claim 1 , wherein the relative traffic volume is calculated as 8 times bidirectional internet protocol (IP) level 3 traffic volume through the authoritative source, in bytes, divided by maximum nominal speed of the authoritative source, in bits per second, multiplied by a data collection interval.
8. The method of claim 1 , wherein the relative packet rate values are calculated as a maximum nominal speed of the authoritative source, in bits per second, divided by 8 multiplied by average network packet size during a data collection interval, in bytes.
9. The method of claim 1 , wherein the further processing includes performing a change point detection calculation.
10. The method of claim 9 , wherein the change point detection calculation is a cumulative sums algorithm.
11. The method of claim 9 , wherein the change point detection calculation is a wavelet transform algorithm.
12. The method of claim 9 , wherein the change point detection calculation is a support vector machine algorithm.