Multiple machine learning model anomaly detection framework
Anomalies may be detected using a multiple machine learning model anomaly detection framework. A clustering model is trained using an unsupervised machine learning algorithm on a historical anomaly dataset. A plurality of clusters of records are determined by applying the historical anomaly dataset to the clustering model. Then it is determined whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster. The plurality of labels for the plurality of records are updated based on the particular record's cluster classification. Non-pure clusters are determined from among the plurality of clusters based on a purity threshold. A supervised machine learning model is trained for each of the non-pure clusters using the records in the given cluster and the labels for each of those records. Then, predictions of an anomaly are made using the clustering model and the supervised machine learning models.
1. A computer system, comprising:
one or more processors;
one or more machine-readable medium coupled to the one or more processors and storing computer program code comprising sets of instructions executable by the one or more processors to:
obtain a historical anomaly dataset comprising a plurality of records and a plurality of labels for each record of the plurality of records, each label indicating whether a particular record is normal or an anomaly;
train a clustering model using an unsupervised machine learning algorithm on the historical anomaly dataset;
determine a plurality of clusters of records by applying the historical anomaly dataset to the clustering model;
determine whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster, the determination of the anomaly-type based on a number of anomaly records in the given cluster being greater than a number of normal records in the given cluster, the determination of the normal-type based on the number of normal records in the given cluster being greater than the number of anomaly records in the given cluster;
update the plurality of labels for the plurality of records based on the particular record's cluster classification being the anomaly-type or the normal-type;
determine one or more non-pure clusters from among the plurality of clusters based on a percentage of normal-labeled records in a normal-type cluster being less than a predetermined purity threshold or based on a percentage of anomaly-labeled records in an anomaly-type cluster being less than the purity threshold;
train a supervised machine learning model for each of the one or more non-pure clusters using the records in the given cluster and the labels for each of those records;
obtain a new dataset comprising one or more new records;
identify the particular cluster of the plurality of clusters that each of the new records belongs to by applying the new dataset to the clustering model; and
predict whether the new records belonging to non-pure clusters are an anomaly by applying the new records to the supervised machine learning model for the corresponding non-pure cluster that they belong to.
2. The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
reduce a dimensionality of the historical anomaly dataset using a feature projection algorithm to obtain a reduced dimensionality historical anomaly dataset, wherein the training of the clustering model used the reduced dimensionality historical anomaly dataset.
3. The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
predict that the new records belonging to pure clusters are an anomaly if they belong to an anomaly-type cluster.
4. The computer system of claim 1 , wherein the labeled historical anomaly dataset comprises categorical data, and wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
identify one or more categorical features in the categorical data;
build a categorical data encoder for each of the one or more categorical features; and
transform values of the categorical data into numerical values using the categorical data encoders.
5. The computer system of claim 1 , wherein the labeled historical anomaly dataset comprises numerical data, and wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
identify one or more numerical features in the numerical data;
build a data standardizer for each of the one or more numerical features; and
transform values of the numerical data for each of numerical features to have a mean of zero and a variance of one.
6. The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
derive engineered features for each of the one or more non-pure clusters.
7. The computer system of claim 1 , wherein the unsupervised machine learning algorithm used to train the clustering model is a gaussian mixture modeling algorithm.
8. A non-transitory computer-readable medium storing computer program code comprising sets of instructions to:
obtain a historical anomaly dataset comprising a plurality of records and a plurality of labels for each record of the plurality of records, each label indicating whether a particular record is normal or an anomaly;
train a clustering model using an unsupervised machine learning algorithm on the historical anomaly dataset;
determine a plurality of clusters of records by applying the historical anomaly dataset to the clustering model;
determine whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster, the determination of the anomaly-type based on a number of anomaly records in the given cluster being greater than a number of normal records in the given cluster, the determination of the normal-type based on the number of normal records in the given cluster being greater than the number of anomaly records in the given cluster;
update the plurality of labels for the plurality of records based on the particular record's cluster classification being the anomaly-type or the normal-type;
determine one or more non-pure clusters from among the plurality of clusters based on a percentage of normal-labeled records in a normal-type cluster being less than a predetermined purity threshold or based on a percentage of anomaly-labeled records in an anomaly-type cluster being less than the purity threshold;
train a supervised machine learning model for each of the one or more non-pure clusters using the records in the given cluster and the labels for each of those records;
obtain a new dataset comprising one or more new records;
identify the particular cluster of the plurality of clusters that each of the new records belongs to by applying the new dataset to the clustering model; and
predict whether the new records belonging to non-pure clusters are an anomaly by applying the new records to the supervised machine learning model for the corresponding non-pure cluster that they belong to.
9. The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
reduce a dimensionality of the historical anomaly dataset using a feature projection algorithm to obtain a reduced dimensionality historical anomaly dataset, wherein the training of the clustering model used the reduced dimensionality historical anomaly dataset.
10. The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
predict that the new records belonging to pure clusters are an anomaly if they belong to an anomaly-type cluster.
11. The non-transitory computer-readable medium of claim 8 , wherein the labeled historical anomaly dataset comprises categorical data, and wherein the computer program code further comprises sets of instructions to:
identify one or more categorical features in the categorical data;
build a categorical data encoder for each of the one or more categorical features; and
transform values of the categorical data into numerical values using the categorical data encoders.
12. The non-transitory computer-readable medium of claim 8 , wherein the labeled historical anomaly dataset comprises numerical data, and wherein the computer program code further comprises sets of instructions to:
identify one or more numerical features in the numerical data;
build a data standardizer for each of the one or more numerical features; and
transform values of the numerical data for each of numerical features to have a mean of zero and a variance of one.
13. The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
derive engineered features for each of the one or more non-pure clusters.
14. The non-transitory computer-readable medium of claim 8 , wherein the unsupervised machine learning algorithm used to train the clustering model is a gaussian mixture modeling algorithm.
15. A computer-implemented method, comprising:
obtaining a historical anomaly dataset comprising a plurality of records and a plurality of labels for each record of the plurality of records, each label indicating whether a particular record is normal or an anomaly;
training a clustering model using an unsupervised machine learning algorithm on the historical anomaly dataset;
determining a plurality of clusters of records by applying the historical anomaly dataset to the clustering model;
determining whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster, the determination of the anomaly-type based on a number of anomaly records in the given cluster being greater than a number of normal records in the given cluster, the determination of the normal-type based on the number of normal records in the given cluster being greater than the number of anomaly records in the given cluster;
updating the plurality of labels for the plurality of records based on the particular record's cluster classification being the anomaly-type or the normal-type;
determining one or more non-pure clusters from among the plurality of clusters based on a percentage of normal-labeled records in a normal-type cluster being less than a predetermined purity threshold or based on a percentage of anomaly-labeled records in an anomaly-type cluster being less than the purity threshold;
training a supervised machine learning model for each of the one or more non-pure clusters using the records in the given cluster and the labels for each of those records;
obtaining a new dataset comprising one or more new records;
identifying the particular cluster of the plurality of clusters that each of the new records belongs to by applying the new dataset to the clustering model; and
predicting whether the new records belonging to non-pure clusters are an anomaly by applying the new records to the supervised machine learning model for the corresponding non-pure cluster that they belong to.
16. The computer-implemented method of claim 15 , further comprising:
reducing a dimensionality of the historical anomaly dataset using a feature projection algorithm to obtain a reduced dimensionality historical anomaly dataset, wherein the training of the clustering model used the reduced dimensionality historical anomaly dataset.
17. The computer-implemented method of claim 15 , further comprising:
predicting that the new records belonging to pure clusters are an anomaly if they belong to an anomaly-type cluster.
18. The computer-implemented method of claim 15 , wherein the labeled historical anomaly dataset comprises categorical data, the method further comprising:
identifying one or more categorical features in the categorical data;
building a categorical data encoder for each of the one or more categorical features; and
transforming values of the categorical data into numerical values using the categorical data encoders.
19. The computer-implemented method of claim 15 , wherein the labeled historical anomaly dataset comprises numerical data, the method further comprising:
identifying one or more numerical features in the numerical data;
building a data standardizer for each of the one or more numerical features; and
transforming values of the numerical data for each of numerical features to have a mean of zero and a variance of one.
20. The computer-implemented method of claim 15 , wherein the unsupervised machine learning algorithm used to train the clustering model is a gaussian mixture modeling algorithm.