Machine learning model evaluation in cyber defense
Testing machine learning sensors by adding obfuscated training data to test data, and performing real time model fit analysis on live network traffic to determine whether to retrain.
1. A process for improving a machine learning sensor for defending a computer network, comprising:
obtaining samples of normal network traffic from said network;
providing samples of cyber attacks from said network or from a repository of cyber attacks, wherein said samples of cyber attacks constitute known attacks;
sample classifying each of said samples as either said normal network traffic or said known attacks, to create ground truths for said samples;
splitting said samples into a training set and a test set, with each of said sets containing samples of said normal network traffic and of said known attacks;
using a model generating algorithm to generate a variety of models for distinguishing between said normal network traffic and said known attacks in said training set;
obfuscating a portion of samples of said known attacks in said training set to create obfuscated attack samples;
adding said obfuscated attack samples to said test set to form an enhanced test set;
performing statistical analysis on performance of said models with said enhanced test set to determine intrusion detection capability error;
selecting one of said models that optimizes a desired model parameter as an optimal model for attack classifying between said normal network traffic, said known attacks, said obfuscated attack samples, and unknown attacks that are unrelated to said training set; and
installing said optimal model in said sensor to determine model fit using live data from real-time network traffic on said computer network, to determine more accurately whether and when to react to intrusion detection alert logs;
wherein said intrusion detection capability error is the sum of the following:
ratio of misclassified known attacks to total misclassified attacks, multiplied by test error of known attacks;
ratio of misclassified unknown attacks to total misclassified attacks, multiplied by test error of unknown attacks; and
ratio of misclassified obfuscated attack samples to total misclassified attacks, multiplied by test error of obfuscated attack samples.