Multiple machine learning model anomaly detection framework
Anomalies may be detected using a multiple machine learning model anomaly detection framework. A clustering model is trained using an unsupervised machine learning algorithm on a historical anomaly dataset. A plurality of clusters of records are determined by applying the historical anomaly dataset to the clustering model. Then it is determined whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster. The plurality of labels for the plurality of records are updated based on the particular record's cluster classification. Non-pure clusters are determined from among the plurality of clusters based on a purity threshold. A supervised machine learning model is trained for each of the non-pure clusters using the records in the given cluster and the labels for each of those records. Then, predictions of an anomaly are made using the clustering model and the supervised machine learning models.
1 . A computer system, comprising:
one or more processors;
one or more machine-readable medium coupled to the one or more processors and storing computer program code comprising sets of instructions executable by the one or more processors to:
determine a plurality of clusters of records by applying a historical anomaly dataset to a trained clustering model for determining whether a particular record is normal or an anomaly;
determine whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster, the determination of the anomaly-type based on a number of anomaly records in a given cluster, the determination of the normal-type based on the number of normal records in the given cluster;
update a plurality of labels for each of a plurality of records based on a particular record's cluster classification being the anomaly-type or the normal-type;
determine one or more non-pure clusters from among the plurality of clusters based on comparing the plurality of labels in a particular cluster to a predetermined purity threshold;
obtain a new dataset comprising one or more new records;
identify the particular cluster of the plurality of clusters that each of the new records belongs to by applying the new dataset to the trained clustering model; and
predict whether the new records belonging to non-pure clusters are an anomaly by applying the new records to a trained supervised machine learning model for the corresponding non-pure cluster that they belong to.
2 . The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
reduce a dimensionality of the historical anomaly dataset using a feature projection algorithm to obtain a reduced dimensionality historical anomaly dataset, wherein a training of the trained clustering model used the reduced dimensionality historical anomaly dataset.
3 . The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
predict that the new records belonging to pure clusters are an anomaly if they belong to an anomaly-type cluster.
4 . The computer system of claim 1 , wherein the trained clustering model is trained using a labeled historical anomaly dataset comprising categorical data, and wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
identify one or more categorical features in the categorical data;
build a categorical data encoder for each of the one or more categorical features; and
transform values of the categorical data into numerical values using the categorical data encoders.
5 . The computer system of claim 1 , wherein the trained clustering model is trained using a labeled historical anomaly dataset comprising numerical data, and wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
identify one or more numerical features in the numerical data;
build a data standardizer for each of the one or more numerical features; and
transform values of the numerical data for each of numerical features to have a mean of zero and a variance of one.
6 . The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
derive engineered features for each of the one or more non-pure clusters.
7 . The computer system of claim 1 , wherein the trained clustering model is trained using a gaussian mixture modeling algorithm.
8 . A non-transitory computer-readable medium storing computer program code comprising sets of instructions to:
determine a plurality of clusters of records by applying a historical anomaly dataset to a trained clustering model for determining whether a particular record is normal or an anomaly;
determine whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster, the determination of the anomaly-type based on a number of anomaly records in a given cluster, the determination of the normal-type based on the number of normal records in the given cluster;
update a plurality of labels for each of a plurality of records based on a particular record's cluster classification being the anomaly-type or the normal-type;
determine one or more non-pure clusters from among the plurality of clusters based on comparing the plurality of labels in a particular cluster a predetermined purity threshold;
obtain a new dataset comprising one or more new records;
identify the particular cluster of the plurality of clusters that each of the new records belongs to by applying the new dataset to the trained clustering model; and
predict whether the new records belonging to non-pure clusters are an anomaly by applying the new records to a trained supervised machine learning model for the corresponding non-pure cluster that they belong to.
9 . The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
reduce a dimensionality of the historical anomaly dataset using a feature projection algorithm to obtain a reduced dimensionality historical anomaly dataset, wherein a training of the trained clustering model used the reduced dimensionality historical anomaly dataset.
10 . The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
predict that the new records belonging to pure clusters are an anomaly if they belong to an anomaly-type cluster.
11 . The non-transitory computer-readable medium of claim 8 , wherein the trained clustering model is trained using a labeled historical anomaly dataset comprising categorical data, and wherein the computer program code further comprises sets of instructions to:
identify one or more categorical features in the categorical data;
build a categorical data encoder for each of the one or more categorical features; and
transform values of the categorical data into numerical values using the categorical data encoders.
12 . The non-transitory computer-readable medium of claim 8 , wherein the trained clustering model is trained using a labeled historical anomaly dataset comprising numerical data, and wherein the computer program code further comprises sets of instructions to:
identify one or more numerical features in the numerical data;
build a data standardizer for each of the one or more numerical features; and
transform values of the numerical data for each of numerical features to have a mean of zero and a variance of one.
13 . The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
derive engineered features for each of the one or more non-pure clusters.
14 . The non-transitory computer-readable medium of claim 8 , wherein the trained clustering model is trained using a gaussian mixture modeling algorithm.
15 . A computer-implemented method, comprising:
determining a plurality of clusters of records by applying a historical anomaly dataset to a trained clustering model for determining whether a particular record is normal or an anomaly;
determining whether each cluster of the plurality of clusters is an anomaly-type cluster or a normal-type cluster, the determination of the anomaly-type based on a number of anomaly records in a given cluster, the determination of the normal-type based on the number of normal records in the given cluster;
updating a plurality of labels for each of a plurality of records based on a particular record's cluster classification being the anomaly-type or the normal-type;
determining one or more non-pure clusters from among the plurality of clusters based on comparing the plurality of labels in a particular cluster to a predetermined purity threshold;
obtaining a new dataset comprising one or more new records;
identifying the particular cluster of the plurality of clusters that each of the new records belongs to by applying the new dataset to the trained clustering model; and
predicting whether the new records belonging to non-pure clusters are an anomaly by applying the new records to a trained supervised machine learning model for the corresponding non-pure cluster that they belong to.
16 . The computer-implemented method of claim 15 , further comprising:
reducing a dimensionality of the historical anomaly dataset using a feature projection algorithm to obtain a reduced dimensionality historical anomaly dataset, wherein a training of the clustering model used the reduced dimensionality historical anomaly dataset.
17 . The computer-implemented method of claim 15 , further comprising:
predicting that the new records belonging to pure clusters are an anomaly if they belong to an anomaly-type cluster.
18 . The computer-implemented method of claim 15 , wherein the trained clustering model is trained using a labeled historical anomaly dataset comprising categorical data, the method further comprising:
identifying one or more categorical features in the categorical data;
building a categorical data encoder for each of the one or more categorical features; and
transforming values of the categorical data into numerical values using the categorical data encoders.
19 . The computer-implemented method of claim 15 , wherein the trained clustering model is trained using a labeled historical anomaly dataset comprising numerical data, the method further comprising:
identifying one or more numerical features in the numerical data;
building a data standardizer for each of the one or more numerical features; and
transforming values of the numerical data for each of numerical features to have a mean of zero and a variance of one.
20 . The computer-implemented method of claim 15 , wherein the trained clustering model is trained using a gaussian mixture modeling algorithm.