System for detecting anomalous network patterns based on analyzing network traffic data and method thereof
A system for detecting anomalous network patterns based on analyzing network traffic data and method thereof are disclosed. The system comprises a flow aggregator subsystem, a flow feature generation subsystem, an anomaly detection subsystem, a dynamic score subsystem, and a calibration subsystem. The system is configured to receive the network traffic data from one or more network endpoints for aggregating the network traffic data into network flow data. The system is configured to generate one or more flow features for each packet associated with the aggregated network traffic data based on a rolling window-based analysis of the one or more attributes. The system is configured to analyze the one or more flow features by utilizing one or more deep-learning models to detect the one or more anomalous network patterns.
1 . A computer-implemented system for detecting one or more anomalous network patterns based on analyzing network traffic data, comprising:
one or more servers configured with one or more hardware processors;
a memory unit coupled to the one or more servers, wherein the memory unit comprises a set of program instructions in form of a plurality of subsystems, configured to be executed by the one or more servers, wherein the plurality of subsystems comprises:
a flow aggregator subsystem configured to receive the network traffic data from one or more network endpoints for aggregating the network traffic data into network flow data,
the network flow data configured with one or more attributes;
a flow feature generation subsystem configured to generate one or more flow features for each packet associated with the aggregated network traffic data based on a rolling window-based analysis of the one or more attributes;
an anomaly detection subsystem configured to analyze the one or more flow features by utilizing one or more deep-learning models to detect the one or more anomalous network patterns of one or more anomalies in the network traffic data thereby assigning a first-stage numerical score to the one or more anomalies; and
a dynamic score subsystem configured to convert the first-stage numerical score into a second-stage numerical score within a pre-defined range using time-series methods based on a dynamically altered threshold score using a cumulative distribution of the first-stage numerical score to detect the one or more anomalous network patterns derived from the analysis of network traffic data,
wherein the flow feature generation subsystem configured with time-series methods, and
the time-series methods are configured to compute smoothed estimates of averages and variances for the one or more flow features over pre-defined time windows, and
the time-series methods comprise at least one of: exponential-weighted smoothing, and on-line cumulative distribution updating, and wherein
the dynamic score subsystem is configured with an alert-generating module, and
the alert-generating module is configured to generate an outlier alert if the second-stage numerical score exceeds the dynamically altered threshold score, indicating a presence of the one or more anomalous network patterns.
2 . The computer-implemented system of claim 1 , wherein the network traffic data comprises at least one of: layer-3 packets, layer-4 Transmission Control Protocol (TCP) flows, layer-4 User Datagram Protocol (UDP) flows, and layer-7 application-level payloads.
3 . The computer-implemented system of claim 1 , wherein the one or more attributes comprises at least one of: source Internet Protocol (IP) addresses, destination Internet Protocol (IP) addresses, destination ports, transfer protocols, flow bytes, flow packets, and timestamps.
4 . The computer-implemented system of claim 1 , wherein the one or more flow features comprises at least one of: source duration, source bytes, source packets, destination duration, destination bytes, and destination packets.
5 . The computer-implemented system of claim 1 , wherein the anomaly detection subsystem procedures configured with an embedding layer,
the embedding layer is configured to segregate numerical state variables and categorical flow labels associated with the one or more flow features.
6 . The computer-implemented system of claim 1 , wherein one or more deep-learning models is configured with a Leaky Rectified Linear Unit (leaky-ReLU),
the Leaky Rectified Linear Unit (leaky-ReLU) configured to perform at least one of an: encoding and decoding of the one or more flow features, to provide an optimal characteristic between at least one of: diverse traffic types, the numerical state variables, and the categorical flow labels to detect the one or more anomalous network patterns.
7 . The computer-implemented system of claim 1 , wherein the anomalies detection subsystem is configured to utilize simulated traffic data in combination with real-time network traffic data, for offline training the one or more deep-learning models,
the one or more deep-learning models is a multi-layer autoencoder neural network.
8 . The computer-implemented system of claim 1 , wherein the pre-defined range for the second-stage numerical score generated by the dynamic score subsystem is configured between 0 and 100,
the dynamic score subsystem is configured to update the pre-defined range of the second-stage numerical score based on the first-stage numerical score and a frequency of detection of the one or more anomalies.
9 . The computer-implemented system of claim 1 , wherein the plurality of subsystems comprises a calibration subsystem,
the calibration subsystem is configured to continuously monitor and update one or more system parameters based on one or more systemwide average metrics,
the one or more system parameters comprises at least one of: the dynamically altered threshold score, weights, and biases of the one or more deep-learning models, and window sizes for the time-series methods; and
the one or more systemwide average metrics comprises at least one of: the source Internet Protocol (IP) addresses, the destination Internet Protocol (IP) addresses, total bytes, packets, flows, average network latency, throughput, and error rates.
10 . The computer-implemented system of claim 1 , wherein the second-stage numerical score and the one or more flow features are configured to store in an elasticsearch database,
the elasticsearch database is configured with a data visualization dashboard to provide various visualization tools, including at least one of: histograms, line charts, pie charts, and heatmaps to visualize the second-stage numerical score and the one or more flow features.
11 . A computer-implemented method for detecting one or more anomalous network patterns based on analyzing network traffic data, comprising:
aggregating, by one or more servers, the network traffic data received from one or more network endpoints into network flow data,
the network flow data configured with one or more attributes;
generating, by the one or more servers, one or more flow features for each packet associated with the aggregated network traffic data based on a rolling window-based analysis of the one or more attributes;
analyzing, by the one or more servers, the one or more flow features by utilizing one or more deep-learning models to detect the one or more anomalous network patterns of one or more anomalies in the network traffic data to assign a first-stage numerical score to the one or more anomalies; and
converting, by the one or more servers, the first-stage numerical score into a second-stage numerical score within a pre-defined range using time-series methods based on a dynamically altered threshold score using a cumulative distribution of the first-stage numerical score to detect the one or more anomalous network patterns derived from the analysis of network traffic data
wherein generating the one or more flow features are configured with time-series methods, and
the time-series methods are configured to compute smoothed estimates of averages and variances for the one or more flow features over pre-defined time windows, and
the time-series methods comprise at least one of: exponential-weighted smoothing, and on-line cumulative distribution updating, and wherein
the one or more servers are configured with an alert-generating module, and
the alert-generating module is configured to generate an outlier alert if the second-stage numerical score exceeds the dynamically altered threshold score, indicating a presence of the one or more anomalous network patterns.
12 . The computer-implemented method of claim 11 , comprising:
visualizing, by the one or more servers, through a data visualization dashboard the second-stage numerical score, and the one or more flow features stored in an elasticsearch database.
13 . The computer-implemented method of claim 11 , comprising:
updating, by the one or more servers, one or more system parameters based on one or more systemwide average metrics;
the one or more system parameters comprises at least one of: the dynamically altered threshold score, weights, and biases of the one or more deep-learning models procedures, and window sizes for time-series methods; and
the one or more systemwide average metrics comprises at least one of: source Internet Protocol (IP) addresses, destination Internet Protocol (IP) addresses, total bytes, packets, flows, average network latency, throughput, and error rates.
14 . The computer-implemented method of claim 11 , wherein the network traffic data comprises at least one of: layer-3 packets, layer-4 TCP/UDP flows, and layer-7 application-level payloads.
15 . The computer-implemented method of claim 11 , wherein the one or more attributes comprises at least one of: source Internet Protocol (IP) addresses, destination Internet Protocol (IP) addresses, destination ports, transfer protocols, flow bytes, flow packets, and timestamps.
16 . The computer-implemented method of claim 11 , wherein the one or more flow features comprises at least one of: source duration, source bytes, source packets, destination duration, destination bytes, and destination packets.
17 . The computer-implemented method of claim 11 , wherein the pre-defined range for the second-stage numerical score generated by the dynamic score subsystem is configured between 0 and 100, and
the dynamic score subsystem is configured to update the pre-defined range of the second-stage numerical score based on the first-stage numerical score and a frequency of detection of the one or more anomalies.
18 . A non-transitory computer-readable storage medium having programmable instructions stored therein, that when executed by one or more servers, cause the one or more servers to:
aggregating network traffic data received from one or more network endpoints into network flow data,
the network flow data configured with one or more attributes;
generating one or more flow features for each packet associated with the aggregated network traffic data based on a rolling window-based analysis of the one or more attributes;
analyzing the one or more flow features by utilizing one or more deep-learning models to detect the one or more anomalous network patterns of one or more anomalies in the network traffic data to assign a first-stage numerical score to the one or more anomalies; and
converting the first-stage numerical score into a second-stage numerical score within a pre-defined range using time-series methods based on a dynamically altered threshold score using a cumulative distribution of the first-stage numerical score to detect the one or more anomalous network patterns derived from the analysis of network traffic data,
wherein generating the one or more flow features are configured with time-series methods, and
the time-series methods are configured to compute smoothed estimates of averages and variances for the one or more flow features over pre-defined time windows, and
the time-series methods comprise at least one of: exponential-weighted smoothing, and on-line cumulative distribution updating, and wherein
the one or more servers are configured with an alert-generating module, and
the alert-generating module is configured to generate an outlier alert if the second-stage numerical score exceeds the dynamically altered threshold score, indicating a presence of the one or more anomalous network patterns.