Machine learning classification of encrypted network traffic
A system for labelling and classification of encrypted network traffic is disclosed. The system employs a Labeler, having a semi-supervised machine learning module for semi-automated labeling of encrypted network traffic, with an initial involvement of a human-in-the-loop intelligence for rapid training of the Labeler. The Labeler produces a labeled training set of encrypted network traffic flows. The system further includes a Modeler having a genetic algorithm module, for automatically selecting a list of network traffic features for further use in real-time classification of the encrypted network traffic, and outputting a corresponding classification model. The system further includes a Classifier for real-time classification of the encrypted network traffic using the classification model. Corresponding methods for labeling and classifying the encrypted network traffic are also provided.
1 . A method of classifying encrypted dataflows in a data network, the method comprising:
employing a hardware network appliance incorporating a machine-learning mechanism to capture a seed set of encrypted dataflows and extract, from each encrypted dataflow, respective metadata;
obtaining from said respective metadata measurable characteristics of said each encrypted dataflow;
initializing a count and a set of approved labels as an empty set;
repeatedly, until a completion condition is met, performing a training procedure of:
increasing said count;
generating for a randomly selected encrypted dataflow of said seed set a respective set of label suggestions based on respective measurable characteristics;
where said count is an integer multiple of a specified spacing integer and said set of approved labels contains at least a predetermined number of approved labels, training a semi-supervised machine-learning model to generate an additional label suggestion, based on said set of approved labels, to be added to said respective set of label suggestions;
determining whether any label suggestion of said respective set of label suggestions is a valid label suggestion to be appended to said set of approved labels while removing said randomly selected encrypted dataflow from said seed set;
and
continually receiving requests to classify fresh dataflows and producing respective classifications using said semi-supervised machine-learning model;
thereby facilitating enhanced operation and security of the data network.
2 . The method of claim 1 , wherein said generating is based on comparing measurable dataflow characteristics, determined through detection of encrypted dataflow patterns, with characteristics of a specified set of recognized dataflow classifications.
3 . The method of claim 1 , further comprising appending said respective classifications to said set of approved labels during successive epochs of prescribed durations:
thereby maintaining continuously updated labeled dataflows.
4 . The method of claim 1 , wherein said completion condition is one of:
a number of remaining encrypted dataflows of said seed set of encrypted dataflows is less than a predetermined lower bound; and
a succession of a predetermined number of executing said performing said training procedure fails to produce any further approved label.
5 . The method of claim 2 , wherein said measurable encrypted dataflow characteristics comprise:
a range of values, a mean value, and a standard deviation of a packet size;
a range of values, a mean value, and a standard deviation of packet inter-arrival time;
a measure of randomness of packet sizes; and
a measure of randomness of packet inter-arrival time.
6 . The method of claim 2 , wherein said recognized dataflow classifications comprise a set of homogeneous dataflows, and predefined composite dataflows each formed as a multiplex of a respective subset of the set of homogeneous dataflows, the homogeneous dataflows comprising one or more of the following:
an audio stream; an audio conversation; a one-directional text; a two directional text; a file transfer; an e-mail message; a point-to-point communication; a web browsing session; a low-resolution video-stream of a first type; and a high-definition video stream.
7 . The method of claim 2 , further comprising producing a further label suggestion based on using an unsupervised learning algorithm.
8 . The method of claim 2 , further comprising determining said respective set of label suggestions, based on respective measurable characteristics, according to a set of rules.
9 . The method of claim 1 , further comprising using a label spreading algorithm for training said semi-supervised machine learning model.
10 . The method of claim 1 , further comprising performing a two-view co-training process for training said semi-supervised machine learning model employing a first labeler based on packet-size parameters of encrypted packets, and a second labeler based on packet inter-arrival time parameters of encrypted packets.
11 . The method of claim 10 , further comprising:
supplying training data to the first labeler comprising cumulative labeled encrypted dataflows determined at the first labeler in addition to switched labeled encrypted dataflows of a respective most-recent subset of a predetermined number of encrypted dataflows of said seed set determined at the second labeler; and
supplying training data to the second labeler comprising cumulative labeled encrypted dataflows determined at the second labeler in addition to switched labeled encrypted dataflows of a respective most-recent subset of said predetermined number of encrypted dataflows of said seed set determined at the first labeler.
12 . The method of claim 10 , further comprising:
supplying training data to the first labeler comprising cumulative labeled encrypted dataflows determined at the first labeler in addition to switched cumulative labeled encrypted dataflows determined at the second labeler; and
supplying training data to the second labeler comprising cumulative labeled encrypted dataflows determined at the second labeler in addition to switched cumulative labeled encrypted dataflows determined at the first label.
13 . The method of claim 1 , wherein said seed set of encrypted dataflows excludes detected encrypted dataflows determined to be outliers according to respective criteria.
14 . The method of claim 1 , further comprising:
configuring said network appliance as a geographically distributed appliance, comprising a central training module and multiple operational modules connected in a star configuration to the central training module through said data network;
at said central training module:
executing said processes of repeatedly performing said training procedure; and
distributing said semi-supervised machine-learning model to the operational modules;
and
at said operational modules:
processing requests from respective users for classifying respective fresh encrypted dataflows; and
communicating resulting classifications to said central training module;
thereby realizing classifications based on current labeled encrypted dataflows.
15 . A hardware network appliance for classifying encrypted dataflows in a data network, comprising:
a training component comprising a machine-learning mechanism configured to:
acquire, through a first network interface, a seed set of encrypted dataflows;
extract, from each encrypted dataflow, of said seed set, respective metadata;
obtain from said respective metadata measurable characteristics of said each dataflow;
initialize a count and a set of approved labels as an empty set;
repeatedly, until a completion condition is met:
increase said count;
generate for a randomly selected dataflow of said seed set a respective set of label suggestions based on respective measurable characteristics;
where said count is an integer multiple of a specified spacing integer and said set of approved labels contains at least a predetermined number of approved labels, train a semi-supervised machine learning model to generate an additional label suggestion, based on said set of approved labels, to be added to said respective set of label suggestions;
and
determine whether any label suggestion of said respective set of label suggestions is a valid label suggestion to be appended to said set of approved labels while removing said randomly selected encrypted dataflow from said seed set;
and
an operational component configured to classify fresh encrypted dataflows using said semi-supervised machine-learning model.
16 . The network appliance of claim 15 , wherein said set of label suggestions is based on comparing measurable dataflow characteristics, determined through detection of encrypted dataflow patterns, with characteristics of a specified set of recognized dataflow classifications.
17 . The network appliance of claim 15 , wherein said operational component is further configured to append resulting classifications of said operational component to said set of approved labels of the training component during each of successive epochs of prescribed durations;
thereby maintaining continuously updated training data.
18 . The network appliance of claim 16 , wherein said training component is configured to produce a further label suggestion based on using an unsupervised learning algorithm.
19 . The network appliance of claim 15 , wherein said training component is further configured to use a label spreading algorithm to train said semi-supervised machine learning model.
20 . The network appliance of claim 15 , wherein said training component is further configured to perform a two-view co-training process to train said semi-supervised machine learning model employing a first labeler based on packet-size parameters of encrypted packets and a second labeler based on packet inter-arrival time parameters of encrypted packets.