Detection of anomaly in network flow data
Disclosed is a method 101 to be used on collected network data flow 116 associated with a network 100 ; the method 101 includes: an anomaly-detection operation 103 including: (A) obtaining the collected network data flow 116 ; and (B) performing an iterative principal component analysis on the collected network data flow 116 to detect an anomaly associated with the collected network data flow 116 . The method may be used in a server and a network, and may also be implemented as a non-transitory computer-readable media. A corresponding system for detecting the anomaly in the network flow data is also provided.
1. A method for detecting an anomaly in a network flow data, comprising:
using a processor for:
(a) collecting the network flow data, characterizing performance of a network, within a time interval divided into multiple time-bins, and generating network flow features from the collected network flow data for each time-bin;
(b) generating input network traffic matrix containing information for the network flow features for respective time-bins;
(c) generating a statistical matrix from the input traffic matrix;
(d) applying a principal component analysis to the statistical matrix to determine one or more principal components of the statistical matrix;
(e) determining an anomaly score for each time-bin using the principal components;
(f) identifying one or more time-bins of the input network traffic matrix having highest anomaly scores;
(g) determining mean values for network flow features across all time-bins, excluding the identified time-bins;
(h) replacing values of the network flow features in the identified time-bins with respective determined mean values of said network flow features to form a modified input network traffic matrix;
(i) replacing the input network traffic matrix with the modified input network traffic matrix, and repeating the steps (c) to (h) a predetermined number of times.
2. The method of claim 1 , wherein the statistical matrix is a covariance matrix.
3. The method of claim 1 , wherein the statistical matrix is a correlation matrix.
4. The method of claim 1 , further comprising associating the time-bin of the input network traffic matrix having a highest anomaly score with a potential network cyber-attack.
5. The method of claim 1 , wherein the generating network flow features comprises generating the network flow features selected from the list: number of bytes, number of packets, number of flows, number of source IP addresses, number of destination IP addresses, entropy of a source IP address, entropy of a destination IP address, DNS (domain name system) bytes, ICMP (Internet Control Message Protocol) bytes, IRC (Internet Relay Chat) bytes, Unique Source Port count, Unique Destination Port count.
6. The method of claim 1 , wherein the determining the anomaly score comprises performing a projection of each time-bin with respect to an anomalous space mapping matrix, calculated using the principal components.
7. The method of claim 1 , wherein the determining the anomaly score comprises determining a square prediction error for each time-bin.
8. The method of claim 1 , further comprising determining one or more data flows, identified by at least one attribute, contained within the identified time-bins of the input network traffic matrix.
9. The method of claim 8 , wherein the attribute includes an IP address, a port number of source, or a port number of a destination for the one or more data flows.
10. The method of claim 1 , wherein the step (d) comprises selecting a principal component having the highest eigenvalue.
11. The method of claim 1 , wherein the step (d) comprises selecting two or more principal components having highest values.
12. A system for detecting an anomaly in a network flow data, comprising:
a processor;
a non-transitory computer readable storage medium having computer readable instructions stored thereon for execution by the processor, causing the processor to:
(a) collect the network flow data, characterizing performance of a network, within a time interval divided into multiple time-bins, and generate network flow features from the collected network flow data for each time-bin;
(b) generate input network traffic matrix containing information for the network flow features for respective time-bins;
(c) generate a statistical matrix from the input traffic matrix;
(d) apply a principal component analysis to the statistical matrix to determine one or more principal components of the statistical matrix;
(e) determine an anomaly score for each time-bin using the principal components;
(f) identify time-bins of the input network traffic matrix having highest anomaly scores;
(g) determine mean values for network flow features across all time-bins, excluding the identified time-bins;
(h) replace values of the network flow features in the identified time-bins with respective determined mean values of said network flow features to form a modified input network traffic matrix;
(i) replace the input network traffic matrix with the modified input network traffic matrix, and repeat the steps (c) to (h) a predetermined number of times.
13. The system of claim 12 , wherein the statistical matrix is a covariance matrix.
14. The system of claim 12 , wherein the statistical matrix is a correlation matrix.
15. The system of claim 12 , wherein the computer readable instructions further cause the processor to associate the time-bin of the input network traffic matrix having a highest anomaly score with a potential network cyber-attack.
16. The system of claim 12 , wherein the computer readable instructions further cause the processor to select the network flow features from the list of: number of bytes, number of packets, number of flows, number of source IP addresses, number of destination IP addresses, entropy of a source IP address, entropy of a destination IP address, DNS (domain name system) bytes, ICMP (Internet Control Message Protocol) bytes, IRC (Internet Relay Chat) bytes, Unique Source Port count, Unique Destination Port count.
17. The system of claim 12 , wherein the computer readable instructions further cause the processor to perform a projection of each feature of the input network traffic matrix and calculate an anomaly score for each time-bin of the input network traffic matrix.
18. The system of claim 12 , wherein the computer readable instructions further cause the processor to determine a square prediction error for each time-bin.
19. The system of claim 12 , wherein the computer readable instructions further cause the processor to determine one or more data flows, identified by at least one attribute, contained within the identified time-bins of the input network traffic matrix.
20. The system of claim 12 , wherein the attribute includes an IP address, a port number of source, or a port number of a destination for the one or more data flows.
21. The system of claim 12 , wherein the computer readable instructions further cause the processor to select a principal component having the highest eigenvalue.
22. The system of claim 12 , wherein the computer readable instructions further cause the processor to select two or more principal components having highest values.