Machine-learning based approach for classification of encrypted network traffic
Systems and methods for a machine-learning based approach for classification of encrypted network traffic data are provided. According to various embodiment of the present disclosure, a network security device receives a stream of packets representing a network flow. Metadata relating to the stream of packets is determined. Application layer payload data of one or more packets of the stream of packets is matched against string patterns and regular expression patterns. Statistics relating to the application layer payload data are collected. The network flow is then classified as being associated with a particular network service of various network services by applying a machine-learning model to the metadata, results of the matching, and the collected statistics.
1 . A method comprising:
receiving, by a processor of a network security device, a stream of packets representing a network flow, the network flow having a handshake stage and an encrypted data exchange stage;
collecting metadata and copying unencrypted payload data to a local memory during the handshake stage;
determining, by the processor, metadata relating to the stream of packets;
matching, by a pattern matching and regular expression matching module of a hardware acceleration sub-system of the network security device, the copied unencrypted application layer payload data of one or more packets of the stream of packets against string patterns and regular expression patterns;
applying, by the processor, a machine learning model to at least a portion of the metadata and results of said matching to generate a preliminary classification output and an associated classification confidence value;
collecting, by the hardware acceleration sub-system, statistics relating to the copied unencrypted application layer payload data when the classification confidence value satisfies a confidence threshold;
collecting, by the hardware acceleration sub-system, additional metadata when the classification confidence value does not satisfy the confidence threshold;
classifying, by the processor, the encrypted data of the network flow as being associated with a particular network service of a plurality of network services by applying a machine-learning model to the metadata, results of said matching, and the collected statistics to generate a classification corresponding to the preliminary classification output when the classification confidence value satisfies the confidence threshold;
determining, by the processor, a model skepticism value associated with the classification, the model skepticism value representing uncertainty of the machine learning model with respect to the classification;
comparing the model skepticism value to a skepticism threshold;
conditionally modifying the machine learning model only when the model skepticism value satisfies the skepticism threshold, wherein the modification comprises selectively retraining or adjusting the machine learning model based on the metadata, results of said matching, or collected statistics; and
reporting a classification result as part of an encrypted data exchange when the classification is successful.
2 . The method of claim 1 , wherein the machine learning model comprises one or more of a decision tree model, a logistic regression model, and a neural network model.
3 . The method of claim 2 , wherein an output layer of the machine learning model outputs a classification of the network flow based on a highest score among predetermined confidence classification scores associated with the decision tree model, the logistic regression model, and the neural network model.
4 . The method of claim 1 , wherein the network security device is also operable to perform intrusion detection functionality and wherein the pattern matching and regular expression matching module is shared by network traffic classification functionality and the intrusion detection functionality.
5 . The method of claim 1 , wherein the metadata comprises any or a combination of a packet size sequence, an arrival interval sequence, an Internet Protocol (IP) family, and a layer four protocol associated with the network flow.
6 . The method of claim 5 , wherein the metadata further comprises any or a combination of a destination port specified by the layer four protocol, Transport Layer Security (TLS) records, and TLS hello message lengths.
7 . The method of claim 5 , wherein the packet size sequence comprises sizes of the application layer payload for a predetermined number of initial packets of the network flow.
8 . A network security device comprising:
a hardware acceleration sub-system configured to:
perform pattern matching and regular expression matching of application layer payload data of received packets against string patterns and regular expression patterns; and
collect statistics relating to the application layer payload data;
a processing resource; and
a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to:
receive a stream of packets representing a network flow, the network flow having a handshake stage and an encrypted data exchange stage;
collect metadata and copy unencrypted application layer payload data to a local memory during the handshake stage;
determine metadata relating to the stream of packets collected on a packet-by-packet basis;
match, by a pattern matching and regular expression matching module of a hardware acceleration sub-system of the network security device, the copied unencrypted application layer payload data of one or more packets of the stream of packets against string patterns and regular expression patterns;
apply a machine learning model to at least a portion of the metadata and results of said matching to generate a preliminary classification output and an associated classification confidence value;
collect, by the hardware acceleration sub-system, statistics relating to the copied unencrypted application layer payload data when a classification confidence value produced by satisfies a confidence threshold;
collect, by the hardware acceleration sub-system, additional metadata when the classification confidence value does not satisfy the confidence threshold;
classify the encrypted data of the network flow as being associated with a particular network service of a plurality of network services by applying a machine-learning model to the metadata, results of said matching, and the collected statistics to generate a classification corresponding to the preliminary classification output when the classification confidence value satisfies the confidence threshold;
determine a model skepticism value associated with the classification, the model skepticism value representing uncertainty of the machine learning model with respect to the classification;
compare the model skepticism value to a skepticism threshold;
conditionally modify the machine learning model only when the model skepticism value satisfies the skepticism threshold, wherein the modification comprises selectively retraining or adjusting the machine learning model based on the metadata, results of said matching, or collected statistics; and
report a classification result as part of an encrypted data exchange when the classification is successful.
9 . The network security device of claim 8 , wherein the instructions that when executed by the processing resource further cause the processing resource to:
make use of the results of to perform pattern matching and regular expression matching to detect intrusion.
10 . The network security device of claim 8 , wherein the machine-learning module comprises a decision tree based model, a logistic regression model, and a neural network model.
11 . The network security device of claim 8 , wherein the hardware acceleration sub-system includes a decision tree co-processor, a multiply-accumulate co-processor, and a lookup table co-processor.
12 . The network security device of claim 10 , wherein the multiply-accumulate co-processor is shared by the logistic regression module and the neural network module.
13 . The network security device of claim 8 , wherein the hardware acceleration sub-system and the processing resource are implemented within a network interface card of the network security device.
14 . The method of claim 1 , wherein the metadata includes at least one of:
a packet size sequence;
an arrival interval sequence;
an internet protocol (IP) family;
a layer four protocol associated with the network flow;
a destination port specified by a layer four protocol associated with the network flow; and
a packet size sequence including a size of an application layer payload.
15 . The method of claim 1 , wherein the statistics are specific to information included within the application layer payload data.
16 . The method of claim 1 , wherein the statistics comprise: a frequency of characters occurring within the application layer payload data.
17 . The method of claim 1 , wherein the statistics comprise: a frequency of character ranges occurring within the application layer payload data.
18 . The method of claim 1 , wherein the classifying yields a classification, and wherein the method further comprises:
reporting, by the processor, the classification.
19 . The network security device of claim 8 , wherein the metadata includes at least one of:
a packet size sequence;
an arrival interval sequence;
an internet protocol (IP) family;
a layer four protocol associated with the network flow;
a destination port specified by a layer four protocol associated with the network flow; and
a packet size sequence including a size of an application layer payload.
20 . The network security device of claim 8 , wherein the statistics are specific to information included within the application layer payload data.
21 . The network security device of claim 8 , wherein the statistics comprise: a frequency of characters occurring within the application layer payload data.
22 . The network security device of claim 8 , wherein the statistics comprise: a frequency of character ranges occurring within the application layer payload data.
23 . The network security device of claim 8 , wherein the hardware acceleration sub-system is further configured to make use of the results of pattern matching and regular expression matching to detect intrusion.
24 . The network security device of claim 8 , wherein the classifying yields a classification further comprises reporting, by the processor, the classification.