System and method for identifying network applications
View Patent ↗Embodiments of the invention provide a framework for traffic classification that bridges the gap between the packet content inspection and the flow-based behavioral analysis techniques. In particular, IP packets and/or IP flows are used as an input, network nodes are associated to specific network applications by leveraging information gathered from the web, and packet-level and/or flow-level signatures are extracted in an off-line fashion using clustering and signature extraction algorithms. The signatures learned are systematically exported to a traffic classifier that uses the newly available signatures to classify applications on-the-fly.
1. A method for profiling network traffic of a network, comprising:
obtaining a first plurality of packet header records from a plurality of flows in the network traffic;
extracting a first plurality of statistical features for each flow from the first plurality of packet header records to form a first server feature matrix of a first server associated with the first plurality of packet header records;
determining, using a computer, a plurality of principal components of the first server feature matrix by factorizing the first server feature matrix;
forming an application space based on the plurality of principal components, wherein the plurality of principal components are orthogonal unit vectors in the application space;
identifying, using the computer, a signature of the first server based on a pattern of the first plurality of packet header records in the application space, wherein the signature is associated with a network application running on the first server; and
determining, using the computer, a second server in the network as running the network application by:
extracting a second plurality of statistical features from a second plurality of packet header records associated with the second server in the network traffic;
placing the second plurality of statistical features in the application space for comparing to the signature to generate a matching level; and
determining the second server as running the network application when the matching level exceeds a pre-determined criterion.
2. The method of claim 1 , wherein obtaining the first plurality of packet header records from the network traffic comprises:
obtaining a plurality of data packets from the network traffic;
identifying an IP address of which percentage occurrences in destination IP address fields in the plurality of data packets exceeds a pre-determined threshold;
identifying an end point of the network associated with the IP address as the first server; and
extracting the first plurality of packet header records from the plurality of data packets based on the IP address of the first server.
3. The method of claim 2 , further comprising:
collecting the plurality of data packets in the network traffic during a first time window, wherein the signature of the first server and the network application are identified during the first time window;
collecting another plurality of data packets in the network traffic during a second time window subsequent to the first time window; and
identifying the second server by analyzing the another plurality of data packets during the second time window, wherein the second server is not identified from the plurality of data packets during the first time window.
4. The method of claim 1 , further comprising:
placing entries of the first server feature matrix in the application space to form the pattern of the first plurality of packet header records; and
determining a centroid and a standard deviation of the entries of the first server feature matrix in the application space,
wherein the signature is defined based on the application space, the centroid, and the standard deviation.
5. The method of claim 1 , further comprising:
identifying a portion of the plurality of principal components by which the first server feature matrix is reconstructed within a pre-determined threshold of deviation;
wherein the application space is formed using the portion of the plurality of principal components.
6. The method of claim 1 , wherein the first plurality of statistical features comprises at least one of average client to server packet size, average server to client packet size, average client to server inter-packet arrival time, average server to client inter-packet arrival time, average client to server packet data rate, average server to client packet data rate, and average flow duration.
7. A non-transitory computer readable medium, embodying instructions when executed by the computer to profile network traffic of a network, the instructions comprising functionality for:
obtaining a plurality of data packets from the network traffic;
identifying an IP address of which percentage occurrences in destination IP address fields in the plurality of data packets exceeds a pre-determined threshold;
identifying an end point of the network associated with the IP address as a first server associated with a plurality of flows in the network traffic; and
extracting a first plurality of packet header records from the plurality of data packets based on the IP address of the first server;
extracting a first plurality of statistical features for each flow from the first plurality of packet header records to form a first server feature matrix of the first server;
determining, using a computer, a plurality of principal components of the first server feature matrix by factorizing the first server feature matrix using singular value decomposition;
forming an application space based on the plurality of principal components, wherein the plurality of principal components are orthogonal unit vectors in the application space;
identifying, using the computer, a signature of the first server based on a pattern of the first plurality of packet header records in the application space, wherein the signature is associated with a network application running on the first server;
determining, using the computer, a second server in the network as running the network application by analyzing a second plurality of packet header records associated with the second server in the network traffic based on the signature.
8. The non-transitory computer readable medium of claim 7 , the instructions when executed by the computer further comprising functionalities for:
placing entries of the first server feature matrix in the application space to form the pattern of the first plurality of packet header records; and
determining a centroid and a standard deviation of the entries of the first server feature matrix in the application space,
wherein the signature is defined based on the application space, the centroid, and the standard deviation.
9. The non-transitory computer readable medium of claim 7 , the instructions when executed by the computer further comprising functionalities for:
identifying a portion of the plurality of principal components by which the first server feature matrix is reconstructed within a pre-determined threshold of deviation;
wherein the application space is formed using the portion of the plurality of principal components.
10. The non-transitory computer readable medium of claim 7 , wherein analyzing the second plurality of packet header records in the network traffic based on the signature comprises:
extracting a second plurality of statistical features from the second plurality of packet header records;
placing the second plurality of statistical features in the application space for comparing to the signature to generate a matching level; and
determining the second server as running the network application when the matching level exceeds a pre-determined criterion.
11. The non-transitory computer readable medium of claim 7 , wherein the first plurality of statistical features comprises at least one of average client to server packet size, average server to client packet size, average client to server inter-packet arrival time, average server to client inter-packet arrival time, average client to server packet data rate, average server to client packet data rate, and average flow duration.
12. The non-transitory computer readable medium of claim 7 , the instructions when executed by the computer further comprising functionalities for:
collecting the plurality of data packets in the network traffic during a first time window, wherein the signature of the first server and the network application are identified during the first time window;
collecting another plurality of data packets in the network traffic during a second time window subsequent to the first time window; and
identifying the second server by analyzing the another plurality of data packets during the second time window, wherein the second server is not identified from the plurality of data packets during the first time window.
13. A system for profiling network traffic of a network, comprising:
a data collector configured to:
obtain a plurality of data packets from the network traffic;
identify an IP address of which percentage occurrences in destination IP address fields in the plurality of data packets exceeds a pre-determined threshold;
identify an end point of the network associated with the IP address as a first server associated with a plurality of flows in the network traffic; and
extract a first plurality of packet header records from the plurality of data packets based on the IP address of the first server;
a statistical analyzer configured to:
extract a first plurality of statistical features for each flow from the first plurality of packet header records to form a first server feature matrix of the first server;
determine a plurality of principal components of the first server feature matrix by factorizing the first server feature matrix using singular value decomposition; and
form an application space based on the plurality of principal components, wherein the plurality of principal components are orthogonal unit vectors in the application space;
a signature generator configured to identify a signature of the first server based on a pattern of the first plurality of packet header records in the application space, wherein the signature is associated with a network application running on the first server; and
a processor and memory storing instructions when executed by the processor comprising functionalities to:
analyze a second plurality of packet header records in the network traffic based on the signature to generate a matching level; and
determine a second server in the network as running the network application based on the matching level.
14. The system of claim 13 , wherein the signature generator is further configured to:
place entries of the first server feature matrix in the application space to form the pattern of the first plurality of packet header records; and
determine a centroid and a standard deviation of the entries of the first server feature matrix in the application space,
wherein the signature is defined based on the application space, the centroid, and the standard deviation.
15. The system of claim 13 , wherein the statistical analyzer is further configured to:
identify a portion of the plurality of principal components by which the first server feature matrix is reconstructed within a pre-determined threshold of deviation;
wherein the application space is formed using the portion of the plurality of principal components.
16. The system of claim 13 , the instructions when executed by the processor further comprising functionalities to:
extract a second plurality of statistical features from the second plurality of packet header records;
place the second plurality of statistical features in the application space for comparing to the signature to generate the matching level; and
determine the second server as running the network application when the matching level exceeds a pre-determined criterion.
17. The system of claim 13 , wherein the first plurality of statistical features comprises at least one of average client to server packet size, average server to client packet size, average client to server inter-packet arrival time, average server to client inter-packet arrival time, average client to server packet data rate, average server to client packet data rate, and average flow duration.
18. The system of claim 13 , wherein the data collector is further configured to:
collect the plurality of data packets in the network traffic during a first time window, wherein the signature of the first server and the network application are identified during the first time window;
collect another plurality of data packets in the network traffic during a second time window subsequent to the first time window; and
identify the second server by analyzing the another plurality of data packets during the second time window, wherein the second server is not identified from the plurality of data packets during the first time window.