Multi-level ensemble classifiers for cybersecurity machine learning applications
Methods, systems, and techniques for producing and using enhanced machine learning models and computer-implemented tools to investigate cybersecurity related data and threat intelligence data are provided. Example embodiments provide an Enhanced Predictive Security System, for building, deploying, and managing applications for evaluating threat intelligence data that can predict malicious domains associated with bad actors before the domains are known to be malicious. In one example, the EPSS comprises one or more components that work together to provide an architecture and a framework for building and deploying cybersecurity threat analysis application, including machine learning algorithms, feature class engines, tuning systems, ensemble classifier engines, and validation and testing engines. These components cooperate and act upon domain data and feature class vectors to create sampled test, training, and validation data and to build model subsets and applications using a trained model library, which stores definitions of each model subset for easy re-instantiation.
1. A method in a computing system, comprising:
determining from a plurality of trained machine learning models stored in a repository, a plurality of model subsets, each model subset including a plurality of model instances sharing a same machine learning algorithm, same model tuning parameters, and same feature vector values as input data, and each of the plurality of model instances of a same model subset trained using a different sampling of training data, wherein each feature vector for each model subset encodes and transforms a subset of feature classes of interest to a respective model subset, and wherein at least two of the plurality of model subsets have different feature vectors;
for each of a plurality of model subsets,
under control of the computing system automatically executing each of the plurality of model instances of each model subset to produce a model output and combining the plurality of model outputs to produce a single model subset output;
forwarding the single model subset output to a plurality of ensemble master classifiers;
under control of at least one of the plurality of ensemble master classifiers, automatically,
receiving as input the single model subset output from at least two different model subsets, wherein each of the single model subset outputs result from processing different internet infrastructure data associated with different feature classes of interest;
initializing weights to correspond to each of the single model subset outputs from the at least two different model subsets thereby defining amount of participation of each of the model subsets participating in the at least one of the plurality of ensemble master classifiers;
performing a machine learning classification on the received input according to current weights associated with each of the single model subsets to generate a score;
optimizing the results of the performed classification by adjusting the weights and re-performing the machine learning classification to regenerate a score until a determined optimization value is satisfied; and
forwarding the score after the optimization value is satisfied as the final score of the ensemble master classifier; and
deploying the resultant optimized at least one of the plurality of ensemble master classifiers as a cybersecurity threat analyzer such that, when the deployed cybersecurity threat analyzer is applied to an input of an unknown domain that is not yet known to be malicious, the cybersecurity threat analyzer automatically predicts and indicates a metric as the final score, without human intervention in the prediction process, whether the unknown domain is malicious.
2. The method of claim 1 wherein the at least one of the plurality of ensemble master classifiers is a deep neural network.
3. The method of claim 1 wherein at least one of the plurality of ensemble master classifiers is an SVM, a Logistic Regression, a Bayesian classifier, a decision tree, a random forest, or gradient boosted tree.
4. The method of claim 1 wherein the same machine learning algorithm shared by model instances of at least one model subset is a one of a generalized linear model, a kernel-based method, a Bayesian-based method, a decision tree, or a deep neural network.
5. The method of claim 1 wherein each ensemble master classifier of the plurality of ensemble master classifiers is associated with an application.
6. The method of claim 5 wherein the application is a phishing detection application or a spam detection application.
7. The method of claim 5 wherein the application is a malware detection application.
8. The method of claim 5 wherein the single model subset output of at least one of the model subsets is forwarded to multiple ensemble master classifiers.
9. The method of claim 1 wherein the single model subset output of each model subset comprises potentially two values wherein a first value is a Boolean classification value or an indication of a likelihood of classification and wherein a second value is an indication of existence of a classification score or confidence in the likelihood of classification.
10. The method of claim 9 wherein the single model subset output of each model subset comprises two values wherein the first value is a Boolean classification value and wherein the second value indicates existence of a score.
11. The method of claim 9 wherein the single model subset output of each model subset comprises two values wherein the first value is a likelihood of classification and wherein the second value indicates confidence in the likelihood of classification.
12. The method of claim 1 wherein the final score comprises a Boolean classification value or an indication of a likelihood of classification.
13. The method of claim 12 wherein the final score comprises a Boolean classification value and indicates existence of a final score.
14. The method of claim 12 wherein the final score comprises a likelihood of classification and indicates confidence in the final score.
15. The method of claim 12 wherein the final score further comprises an indication of existence of a classification score or confidence in the final score.
16. The method of claim 1 wherein the optimizing the results and re-performing the machine learning classification to regenerate a score is performed using gradient descent optimization.
17. The method of claim 1 wherein the internet infrastructure data comprises one or more of domain names, whois information, IP addresses, DNS record data, pDNS activity, on-page HTML content, and SSL certificates.
18. The method of claim 1 wherein at least two of the plurality of model subsets correspond to different internet infrastructure data and correspondingly each of the at least two model subsets correspond to different feature classes of internet infrastructure data.
19. The method of claim 1 wherein at least one of the two different model subsets classifies domain name information.
20. The method of claim 1 , further comprising:
under control of a second ensemble master classifier,
receiving as input the single model subset output from at least one model subset; and
using the single model subset output to generate a final score without iteration through another machine learning algorithm.
21. The method of claim 20 wherein the second ensemble master classifier employs voting, ranking, or bagging to generate the final score.
22. The method of claim 20 wherein the final score comprises a Boolean value or an indication of a likelihood of classification.
23. The method of claim 22 wherein the final score further comprises an indication of existence of or confidence for the final score.
24. The method of claim 1 , further comprising:
storing in the repository metadata associated with each of the plurality of model subsets, the metadata including sufficient information to programmatically create each model subset.
25. The method of claim 24 wherein the metadata associated with each model subset includes an indication of a machine learning algorithm, a set of hyper parameters for tuning the indicated machine learning algorithm, a description of feature class information used to build an associated input feature vector, an indication of a source for training data, and an indication of training data sampling parameters.
26. A computing system configured to automatically classify a domain as spam, malware, or phishing, comprising:
a data repository having stored therein a plurality of trained machine learning models;
a feature class selection and transformation module configured to determine from the data repository a plurality of model subsets, each model subset including a plurality of model instances sharing a same machine learning algorithm, same model tuning parameters, and same feature vector values as input data, and each of the plurality of model instances of a same model subset trained using a different sampling of training data, wherein each feature vector for each model subset encodes and transforms a subset of feature classes of interest to a respective model subset, and wherein at least two of the plurality of model subsets have different feature vectors;
an ensemble classifier engine configured to, for each of a plurality of model subsets,
automatically execute each of the plurality of model instances of each model subset to produce a model output and combining the plurality of model outputs to produce a single model subset output;
forward the single model subset output to a plurality of ensemble master classifiers;
under control of at least one of the plurality of ensemble master classifiers, automatically,
receive as input the single model subset output from at least two different model subsets, wherein each of the single model subset outputs result from processing different internet infrastructure data associated with different feature classes of interest;
initialize weights to correspond to each of the single model subset outputs from the at least two different model subsets thereby defining amount of participation of each of the model subsets participating in the at least one of the plurality of ensemble master classifiers;
perform a machine learning classification on the received input according to current weights associated with each of the single model subsets to generate a score;
optimize the results of the performed classification by adjusting the weights and re-performing the machine learning classification to regenerate a score until a determined optimization value is satisfied; and
forward the score after the optimization value is satisfied as the final score of the ensemble master classifier; and
a cybersecurity threat analyzer configured to deploy the resultant optimized at least one of the plurality of ensemble master classifiers that, upon receiving input of an unknown domain that is not yet known to be malicious, automatically predicts and indicates a metric as the final score, without human intervention in the prediction process, whether the unknown domain is malicious.
27. A computer-readable memory medium containing instructions for controlling a computer processor, when executed, to classify a domain as spam, malware, or phishing by performing a method comprising:
determining from a plurality of trained machine learning models stored in a repository, a plurality of model subsets, each model subset including a plurality of model instances sharing a same machine learning algorithm, same model tuning parameters, and same feature vector values as input data, and each of the plurality of model instances of a same model subset trained using a different sampling of training data, wherein each feature vector for each model subset encodes and transforms a subset of feature classes of interest to a respective model subset, and wherein at least two of the plurality of model subsets have different feature vectors;
for each of a plurality of model subsets,
under control of the computing system automatically executing each of the plurality of model instances of each model subset to produce a model output and combining the plurality of model outputs to produce a single model subset output;
forwarding the single model subset output to a plurality of ensemble master classifiers;
under control of at least one of the plurality of ensemble master classifiers, automatically,
receiving as input the single model subset output from at least two different model subsets, wherein each of the single model subset outputs result from processing different internet infrastructure data associated with different feature classes of interest;
initializing weights to correspond to each of the single model subset outputs from the at least two different model subsets thereby defining amount of participation of each of the model subsets participating in the at least one of the plurality of ensemble master classifiers;
performing a machine learning classification on the received input according to current weights associated with each of the single model subsets to generate a score;
optimizing the results of the performed classification by adjusting the weights and re-performing the machine learning classification to regenerate a score until a determined optimization value is satisfied; and
forwarding the score after the optimization value is satisfied as the final score of the ensemble master classifier; and
deploying the resultant optimized at least one of the plurality of ensemble master classifiers as a cybersecurity threat analyzer such that, when the deployed cybersecurity threat analyzer is applied to an input of an unknown domain that is not yet known to be malicious, the cybersecurity threat analyzer automatically predicts and indicates a metric as the final score, without human intervention in the prediction process, whether the unknown domain is malicious.