System and method for improved end-to-end cybersecurity machine learning and deployment
The presently disclosed subject matter includes an apparatus that receives a dataset with values associated with different digital resources captured from a group of compute devices. The apparatus includes a feature extractor, to generate a set of feature vectors, each feature vector from the set of feature vectors associated with a set of data included in the received dataset. The apparatus uses the set of feature vectors to validate multiple machine learning models trained to determine whether a digital resource is associated with a cyberattack. The apparatus selects at least one active machine learning model and sets the remaining trained machine learning models to operate in an inactive mode. The active machine learning model generates a signal to alert a security administrator, blocks a digital resource from loading at a compute device, or executes other remedial action, upon a determination that the digital resource is associated with a cyberattack.
1. A method, comprising:
generating a plurality of trained machine learning models, each trained machine learning model of the plurality of trained machine learning models trained to classify whether a digital resource is associated with a respective cyberattack of a plurality of cyberattacks;
selecting, based on a first performance metric, a first machine learning model and a second machine learning model from the plurality of trained machine learning models to operate in an in-line mode and a third machine learning model from the plurality of trained machine learning models to operate in an out-of-band mode,
the in-line mode being characterized by generating a first classification on whether the digital resource is associated with any cyberattack of the plurality of cyberattacks and using the first classification in determining remedial actions,
the out-of-band mode being characterized by generating a second classification on whether the digital resource is associated with any cyberattack of the plurality of cyberattacks and without using the second classification in determining a remedial action even when the second classification indicates an association with a cyberattack of the plurality of cyberattacks;
deploying, to one or more compute devices via a network, the first machine learning model, the second machine learning model, and the third machine learning model to classify a plurality of digital resources;
evaluating performance following execution of the first machine learning model, the second machine learning model, and the third machine learning model on the one or more compute devices; and
sending a signal to the one or more compute devices to change, in response to an evaluation that the third machine learning model outperforms the first machine learning model or the second machine learning model based on a second performance metric, a configuration of the first machine learning model or the second machine learning model from the in-line mode to the out-of-band mode and a configuration of the third machine learning model from the out-of-band mode to the in-line mode.
2. The method of claim 1 , wherein when in the out-of-band mode, the second machine learning model classifies the digital resource based on a copy of the digital resource.
3. The method of claim 1 , further including:
receiving, based on an event-based condition, a labeled dataset; and
retraining the plurality of trained machine learning models based on the labeled dataset.
4. The method of claim 3 , wherein the event-based condition is met when a number of feature values in a set of feature values received from a set of data sources reaches a predetermined threshold and the set of feature values have not been used to train, validate, or evaluate the plurality of trained machine learning models.
5. The method of claim 3 , wherein the event-based condition is met when a predetermined threshold number of new types of malware has been detected via one or more data sources.
6. The method of claim 1 , further including causing, after sending the signal to change the configuration of the second machine learning model from the out-of-band mode to the in-line mode, one or more remedial actions from the remedial actions to be executed on the one or more compute devices in response to the second machine learning model classifying the digital resource as associated with a cyberattack from the plurality of cyberattacks.
7. The method of claim 1 , further including: deploying the third machine learning model different from the first machine learning model and the second machine learning model and from the plurality of trained machine learning models to operate in the in-line mode when the first machine learning model and the second machine learning model fail to satisfy a performance threshold.
8. The method of claim 7 , wherein failing to satisfy the performance threshold includes failing at least one of (i) achieving a performance above an accuracy threshold, or (ii) achieving a performance below an error threshold.
9. The method of claim 1 , wherein the one or more compute devices include a plurality of compute devices.
10. The method of claim 1 , further including causing one or more remedial actions from the remedial actions to be executed on the one or more compute devices when a statistically-combined classification value of a classification using the first machine learning model and a classification using the second machine learning model indicates the digital resource is associated with a cyberattack from the plurality of cyberattacks.
11. A method, comprising:
generating a plurality of trained machine learning models, each trained machine learning model of the plurality of trained machine learning models to classify whether a digital resource is associated with a respective cyberattack of a plurality of cyberattacks;
deploying, to one or more compute devices via a network, a first machine learning model and a second machine learning model from the plurality of trained machine learning models and a third machine learning model from the plurality of trained machine learning models, the first machine learning model and the second machine learning model to be executed by the one or more compute devices in an in-line mode to classify whether a plurality of digital resources is associated with any cyberattack from the plurality of cyberattacks and determine remedial actions, the third machine learning model to be executed by the one or more compute devices in an out-of-band mode to classify whether the plurality of digital resources is associated with any cyberattack from the plurality of cyberattacks and not determine any remedial actions even when the third machine learning model classifies the digital resource from the plurality of digital resources being associated with a cyberattack from the plurality of cyberattacks;
sending a signal to the one or more compute devices to change, in response to an evaluation of a performance of the first machine learning model, the second machine learning model, and a performance of the third machine learning model, a configuration of the first machine learning model and the second machine learning model, and a configuration of the third machine learning model between the in-line mode and the out-of-band mode; and
deploying, to the one or more compute devices via the network, a fourth machine learning model from the plurality of trained machine learning models different from the first machine learning model, the second machine learning model, and the third machine learning model to operate in the in-line mode when the first machine learning model, the second machine learning model, and the third machine learning model fail to satisfy a performance threshold.
12. The method of claim 11 , wherein failing to satisfy the performance threshold includes failing at least one of (i) achieving a performance above an accuracy threshold, or (ii) achieving a performance below an error threshold.
13. The method of claim 11 , wherein when in the out-of-band mode, the second machine learning model classifies the digital resource based on a copy of the digital resource.
14. The method of claim 11 , further including:
receiving, based on an event-based condition, a labeled dataset; and
retraining the plurality of trained machine learning models based on the labeled dataset.
15. The method of claim 14 , wherein the event-based condition is met when a number of feature values in a set of feature values received from a set of data sources reaches a predetermined threshold and the set of feature values have not been used to train, validate, or evaluate the plurality of trained machine learning models.
16. The method of claim 14 , wherein the event-based condition is met when a predetermined threshold number of new types of malware has been detected via one or more data sources.
17. The method of claim 11 , further including causing, after sending the signal to change the configuration of the second machine learning model from the out-of-band mode to the in-line mode, one or more remedial actions from the remedial actions to be executed on the one or more compute devices in response to the second machine learning model classifying the digital resources as associated with a cyberattack from the plurality of cyberattacks.
18. The method of claim 11 , further including causing one or more remedial actions from the remedial actions to be executed on the one or more compute devices when a statistically-combined classification value of a classification using the first machine learning model and a classification using the second machine learning model indicates the digital resource is associated with a cyberattack from the plurality of cyberattacks.
19. The method of claim 11 , wherein the one or more compute devices include a plurality of compute devices.
20. An apparatus, comprising:
memory; and
a processor operatively coupled with the memory, the processor configured to:
generate a plurality of trained machine learning models, each trained machine learning model of the plurality of trained machine learning models to classify whether a digital resource associated with a respective cyberattack of a plurality of cyberattacks;
receive a first machine learning model and a second machine learning model from the plurality of trained machine learning models and a third machine learning model from the plurality of trained machine learning models;
execute the first machine learning model and the second machine learning model in an in-line mode and the third machine learning model in an out-of-band mode,
in the in-line mode, executing the first machine learning model and the second machine learning model, generating a first classification on whether the digital resource is associated with any cyberattack of the plurality of cyberattacks and using the first classification in determining remedial actions,
in the out-of-band mode, executing the third machine learning model generating a second classification on whether the digital resource is associated with any cyberattack of the plurality of cyberattacks and without using the second classification in determining a remedial action even when the second classification indicates an association with a cyberattack of the plurality of cyberattacks; and
receive a signal in response to an evaluation of performance of the first machine learning model and the second machine learning model failing to satisfy at least one performance threshold, the signal causing (i) the first machine learning model or the second machine learning model to operate in the out-of-band mode, and ii) the third machine learning model to operate in the in-line mode.