Framework for semi-supervised learning when no labeled data is given
A method, including collecting communication sessions, and generating samples from the sessions. Classifiers are applied to the samples, thereby computing a classifier prediction for each sample, and based on the classifier predictions, respective aggregated predictions are determined for the samples. Based on the classifier and the aggregated predictions, a precision and a hit rate for each classifier and a positive rate are computed, and based on the aggregated predictions, a subset of the samples are selected. Using the selected subset, a model including the classifiers is computed based on the precisions, the hit rates and the positive rate, and the model is applied to the samples, thereby updating the classifier and the aggregate predictions. The steps of computing the precision and the hit rate, selecting the subset, computing the model and applying the model are repeated until meeting a halting condition, and using the model, additional communication sessions are scanned.
1. A method, comprising:
collecting, by a processor, communication sessions comprising data packets transmitted over a network;
generating a plurality of samples from the collected data packets;
applying multiple classifiers to the samples, thereby computing a classifier prediction for each of the samples, each of the predictions indicating whether or not a given communication session indicates malicious activity;
determining, based on the classifier predictions, an aggregated prediction for each of the samples;
computing, based on the classifier predictions and the aggregated predictions, a precision and a hit rate for each of the classifiers and a positive rate;
selecting, based on the aggregated predictions, a subset of the samples;
computing, using the selected subset and based on the precisions and the hit rates of the classifiers and the positive rate, a classification model comprising the classifiers;
applying the classification model to the samples to update the classifier predictions and the aggregate predictions for the samples;
repeating over multiple iterations the steps of computing the precision and the hit rate, selecting the subset, computing the classification model and applying the classification model until a halting condition is met; and
scanning, using the classification model, additional communication sessions comprising additional data packets.
2. The method according to claim 1 , wherein the classifiers comprise weakly-useful predictors defined by one or more domain experts.
3. The method according to claim 1 , and comprising generating, using one or more semi-supervised learning methods, the classifiers.
4. The method according to claim 1 , wherein selecting the subset comprises selecting a plurality of the samples that have predictions with a high confidence above a threshold percentage.
5. The method according to claim 1 , wherein selecting the subset during the iterations comprises selecting a plurality of the samples that have disagreements in their classifier predictions, and comprising receiving manual labels for selected subset, wherein the predictions for the selected subset comprises the manual labels.
6. The method according to claim 5 , and comprising adapting the classifiers to a natural distribution of the samples.
7. The method according to claim 1 , wherein selecting the subset during the iterations comprises computing, for each given sample, a score based on the aggregate prediction for the given sample and the computed hit rates, precisions and positive rate.
8. The method according to claim 1 , wherein computing the classification model comprises computing a Naïve Bayes probability using the classifiers.
9. The method according to claim 1 , wherein computing the classification model comprises computing a score for each of the classifiers by applying a linear regression algorithm or a logistic regression algorithm to the classifiers.
10. The method according to claim 1 , wherein the halting condition comprises detecting a minimal change in the predictions during the multiple iterations.
11. The method according to claim 1 , wherein the halting condition comprises detecting minimal changes in the precisions during the multiple iterations.
12. The method according to claim 1 , wherein the halting condition comprises repeating the steps of computing the precision and the hit rate, selecting the subset, computing the classification model and applying the classification model a specific number of times.
13. The method according to claim 1 , wherein computing the classification model comprises computing parameters for the model that minimizes entropy less mutual information with a concept of the samples.
14. An apparatus, comprising:
a probe; and
at least one processor configured:
collect, from the probe, communication sessions comprising data packets transmitted over a network,
to generate a plurality of samples from the collected data packets,
to apply multiple classifiers to the samples, thereby computing a classifier prediction for each of the samples, each of the predictions indicating whether or not a given communication session indicates malicious activity,
to determine, based on the classifier predictions, an aggregated prediction for each of the samples,
to compute, based on the classifier predictions and the aggregated predictions, a precision and a hit rate for each of the classifiers and a positive rate,
to select, based on the aggregated predictions, a subset of the samples,
to compute, using the selected subset and based on the precisions and the hit rates of the classifiers and the positive rate, a classification model comprising the classifiers,
to apply the classification model to the samples to update the classifier predictions and the aggregate predictions for the samples,
to repeat over multiple iterations the steps of computing the precision and the hit rate, selecting the subset, computing the classification model and applying the classification model until a halting condition is met, and
to scan, using the classification model, additional communication sessions comprising additional data packets.
15. The apparatus according to claim 14 , wherein the classifiers comprise weakly-useful predictors defined by one or more domain experts.
16. The apparatus according to claim 14 , wherein the processor is configured to generate, using one or more semi-supervised learning methods, the classifiers.
17. The apparatus according to claim 14 , wherein the processor is configured to select the subset by selecting a plurality of the samples that have predictions with a with a confidence above a threshold percentage.
18. The apparatus according to claim 14 , wherein the processor is configured to select the subset during the iterations by selecting a plurality of the samples that have disagreements in their classifier predictions, and wherein the processor is configured to receive manual labels for selected subset, wherein the predictions for the selected subset comprises the manual labels.
19. The apparatus according to claim 18 , wherein the processor is configured to adapt the classifiers to a natural distribution of the samples.
20. The apparatus according to claim 14 , wherein the processor is configured to select the subset during the iterations by computing, for each given sample, a score based on the aggregate prediction for the given sample and the computed hit rates, precisions and positive rate.
21. The apparatus according to claim 14 , wherein the processor is configured to compute the classification model by computing a Naïve Bayes probability using the classifiers.
22. The apparatus according to claim 14 , wherein the processor is configured to compute the classification model by computing a score for each of the classifiers by applying a linear regression algorithm or a logistic regression algorithm to the classifiers.
23. The apparatus according to claim 14 , wherein the halting condition comprises detecting a minimal change in the predictions during the multiple iterations.
24. The apparatus according to claim 14 , wherein the halting condition comprises detecting minimal changes in the precisions during the multiple iterations.
25. The apparatus according to claim 14 , wherein the halting condition comprises repeating the steps of computing the precision and the hit rate, selecting the subset, computing the classification model and applying the classification model a specific number of times.
26. The apparatus according to claim 14 , wherein the processor is configured to compute the classification model by computing parameters for the model that minimizes entropy less mutual information with a concept of the samples.
27. A computer software product, the product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer, cause the computer:
to collect communication sessions comprising data packets transmitted over a network;
to generate a plurality of samples from the collected data packets;
to apply multiple classifiers to the samples, thereby computing a classifier prediction for each of the samples, each of the predictions indicating whether or not a given communication session indicates malicious activity;
to determine, based on the classifier predictions, an aggregated prediction for each of the samples;
to compute, based on the classifier predictions and the aggregated predictions, a precision and a hit rate for each of the classifiers and a positive rate;
to select, based on the aggregated predictions, a subset of the samples;
to compute, using the selected subset and based on the precisions and the hit rates of the classifiers and the positive rate, a classification model comprising the classifiers;
to apply the classification model to the samples to update the classifier predictions and the aggregate predictions for the samples;
to repeat over multiple iterations the steps of computing the precision and the hit rate, selecting the subset, computing the classification model and applying the classification model until a halting condition is met; and
to scan, using the classification model, additional communication sessions comprising additional data packets.