Detecting and preventing cyber threats
A system for detecting and preventing cyber threats is disclosed. The system is programmed to collect user activity data across different computer applications and devices and threat intelligence data in a distributed computing environment. From the collected data, the system is programmed to then build a predictive model and a counterfactual reasoning model using machine learning and establish a meta model that fuses the two models together. Given new user activity data, the system is programmed to apply the meta model to generate a threat assessment and a recommendation for updating the distributed environment based on the threat assessment.
1 . A method of detecting or preventing cyber threats in a distributed computing environment, comprising:
obtaining a first machine learning model based on a first training dataset,
the first training dataset having a first plurality of feature vectors and classifications of the first plurality of feature vectors into a first plurality of classes,
each feature vector of the first plurality of feature vectors corresponding to a plurality of types of data, including graph data representing relationships among user accounts, computing devices, and computer applications and time series data representing changes over time,
the first machine learning model generating a first score for specific input data to the first machine learning model indicating how well the specific input data matches the first plurality of feature vectors,
the first score including an aggregate of a plurality of scores for the first plurality of classes,
the first plurality of classes corresponding to at least multiple abnormal conditions;
obtaining a second machine learning model that is a generative AI model based on a second training dataset,
the second training dataset having a second plurality of feature vectors and classifications of the second plurality of feature vectors including a second plurality of classes,
each feature vector of the second plurality of feature vectors corresponding to the plurality of types of data,
the second machine learning model generating a second score for certain input data into the second machine learning model indicating how much the certain input data deviates from the second plurality of feature vectors,
the second plurality of classes corresponding to no anomalous condition;
obtaining fusion logic that blends the first machine learning model and the second machine learning model to generate a third score, comprising using an output of the second machine learning model to determine an input of the first machine learning model;
receiving new data associated with a user account, including particular graph data and particular time series data;
generating a new feature vector from the new data;
applying the fusion logic to the new feature vector to obtain a final output;
determining that the final output satisfies a predetermined condition;
taking an action based on the determining, including preventing, aborting, or rolling back a database transaction,
wherein the method is performed by one or more processors.
2 . The method of claim 1 , the plurality of types of data including contextual data representing narratives in natural language.
3 . The method of claim 1 ,
obtaining the first machine learning model comprising training the first machine learning model,
obtaining the second machine learning model comprising training the second machine learning model.
4 . The method of claim 1 ,
the first machine learning model corresponding to random forest, eXtreme Gradient Boosting (xGBoost), or deep neural network,
the second machine learning model corresponding to variational autoencoder or generative adversarial network.
5 . The method of claim 1 , obtaining the fusion logic comprising executing the first machine learning model and the second machine learning model on a validation dataset separate from the first training dataset and the second training dataset in a Bayesian optimization framework.
6 . The method of claim 1 ,
the fusion logic including a meta model being a weighted ensemble,
applying the fusion logic comprising executing the first machine learning model and the second machine learning model and applying the weighted ensemble to results of the executing.
7 . The method of claim 1 , the generating a new feature vector comprising converting the particular graph data using a random walk-based method or a graph neural network and converting the particular time series data using a recurrent neural network or a transformer that models temporal dynamics.
8 . The method of claim 1 , the new data representing user activity associated with the user account and threat intelligence associated with the distributed computing environment at a given time.
9 . The method of claim 1 ,
obtaining the first machine learning model comprising generating a first set of feature importance scores from the first machine learning model for features of the first plurality of feature vectors,
obtaining the second machine learning model comprising generating a second set of feature importance scores from the second machine learning model for features of the second plurality of feature vectors.
10 . The method of claim 9 ,
the obtaining fusion logic comprising combining the first set of feature importance scores and the second set of feature importance scores into a final set of feature importance scores,
the taking an action comprising generating a recommendation based on the final set of feature importance scores.
11 . The method of claim 1 , the action including rejecting a request for a data access operation or reversing a data access operation.
12 . The method of claim 1 ,
the action including obtaining an assessment of the new data from an administrator device; the method further comprising:
updating the first training dataset or the second training dataset to obtain an updated first training dataset or an updated second training dataset;
retaining the first machine learning model based on the updated first training dataset or the second machine learning model based on the updated second training dataset.
13 . A system for detecting or preventing cyber threats in a distributed computing environment, comprising:
a memory;
one or more processors coupled with the memory and configured to perform:
obtaining a first machine learning model based on a first training dataset,
the first training dataset having a first plurality of feature vectors and classifications of the first plurality of feature vectors into a first plurality of classes,
each feature vector of the first plurality of feature vectors corresponding to a plurality of types of data, including graph data representing relationships among user accounts, computing devices, and computer applications and time series data representing changes over time,
the first machine learning model generating a first score for specific input data to the first machine learning model indicating how well the specific input data matches the first plurality of feature vectors,
the first score including an aggregate of a plurality of scores for the first plurality of classes,
the first plurality of classes corresponding to at least multiple abnormal conditions;
obtaining a second machine learning model that is a generative AI model based on a second training dataset,
the second training dataset having a second plurality of feature vectors and classifications of the second plurality of feature vectors including a second plurality of classes,
each feature vector of the second plurality of feature vectors corresponding to the plurality of types of data,
the second machine learning model generating a second score for certain input data into the second machine learning model indicating how much the certain input data deviates from the second plurality of feature vectors,
the second plurality of classes corresponding to no anomalous condition;
obtaining fusion logic that blends the first machine learning model and the second machine learning model to generate a third score, comprising using an output of the second machine learning model to determine an input of the first machine learning model;
receiving new data associated with a user account, including particular graph data and particular time series data;
generating a new feature vector from the new data;
applying the fusion logic to the new feature vector to obtain a final output;
determining that the final output satisfies a predetermined condition;
taking an action based on the determining, including preventing, aborting, or rolling back a database transaction.
14 . The system of claim 13 , obtaining the fusion logic comprising executing the first machine learning model and the second machine learning model on a validation dataset separate from the first training dataset and the second training dataset in a Bayesian optimization framework.
15 . The system of claim 13 ,
the fusion logic including a meta model being a weighted ensemble,
applying the fusion logic comprising executing the first machine learning model and the second machine learning model and applying the weighted ensemble to results of the executing.
16 . The system of claim 13 , the new data representing user activity associated with the user account and threat intelligence associated with the distributed computing environment at a given time.
17 . The system of claim 13 ,
obtaining the first machine learning model comprising generating a first set of feature importance scores from the first machine learning model for features of the first plurality of feature vectors,
obtaining the second machine learning model comprising generating a second set of feature importance scores from the second machine learning model for features of the second plurality of feature vectors.
18 . The system of claim 17 ,
the obtaining fusion logic comprising combining the first set of feature importance scores and the second set of feature importance scores into a final set of feature importance scores,
the taking an action comprising generating a recommendation based on the final set of feature importance scores.
19 . The system of claim 13 , the action including rejecting a request for a data access operation or reversing a data access operation.
20 . A non-transitory, computer-readable storage medium storing one or more sequences of instructions which when executed cause one or more processor to perform:
obtaining a first machine learning model based on a first training dataset,
the first training dataset having a first plurality of feature vectors and classifications of the first plurality of feature vectors into a first plurality of classes,
each feature vector of the first plurality of feature vectors corresponding to a plurality of types of data, including graph data representing relationships among user accounts, computing devices, and computer applications and time series data representing changes over time,
the first machine learning model generating a first score for specific input data to the first machine learning model indicating how well the specific input data matches the first plurality of feature vectors,
the first score including an aggregate of a plurality of scores for the first plurality of classes,
the first plurality of classes corresponding to at least multiple abnormal conditions;
obtaining a second machine learning model that is a generative AI model based on a second training dataset,
the second training dataset having a second plurality of feature vectors and classifications of the second plurality of feature vectors including a second plurality of classes,
each feature vector of the second plurality of feature vectors corresponding to the plurality of types of data,
the second machine learning model generating a second score for certain input data into the second machine learning model indicating how much the certain input data deviates from the second plurality of feature vectors,
the second plurality of classes corresponding to no anomalous condition;
obtaining fusion logic that blends the first machine learning model and the second machine learning model to generate a third score, comprising using an output of the second machine learning model to determine an input of the first machine learning model;
receiving new data associated with a user account, including particular graph data and particular time series data;
generating a new feature vector from the new data;
applying the fusion logic to the new feature vector to obtain a final output;
determining that the final output satisfies a predetermined condition;
taking an action based on the determining, including preventing, aborting, or rolling back a database transaction.