Unsupervised anomaly detection via supervised methods

1. A method comprising:

training, by a computer system, an unsupervised anomaly detection classifier using an unlabeled training data set comprising a plurality of unlabeled data instances, the training resulting in a trained version of the unsupervised anomaly detection classifier;

classifying, by the computer system, the unlabeled training data set used to train the unsupervised anomaly detection classifier via the trained version of the unsupervised anomaly detection classifier, the classifying comprising, for each unlabeled data instance in the unlabeled training data set:

providing the unlabeled data instance as a query input to the trained version of the unsupervised anomaly detection classifier; and

generating, via the trained version of the unsupervised anomaly detection classifier, an anomaly score for the unlabeled data instance;

constructing, by the computer system, a labeled training data set obtained from the trained version of the unsupervised anomaly detection classifier that is derived from a first subset of unlabeled data instances in the unlabeled training data set whose anomaly scores are below a first threshold and a second subset of unlabeled data instances in the unlabeled training data set whose anomaly scores are above a second threshold, the constructing comprising, for each unlabeled data instance in the unlabeled training data set:

upon determining that the anomaly score for the unlabeled data instance is below the first threshold, adding the unlabeled data instance as a new labeled data instance to the labeled training data set with a class label indicating that the new labeled data instance is normal; and

upon determining that the anomaly score for the unlabeled data instance is above the second threshold, adding the unlabeled data instance as a new labeled data instance to the labeled training data set with a class label indicating that the new labeled data instance is anomalous;

training, by the computer system, a supervised anomaly detection classifier using the labeled training data set obtained from the trained version of the unsupervised anomaly detection classifier, the training resulting in a trained version of the supervised anomaly detection classifier;

receiving an unlabeled query data set comprising another plurality of unlabeled data instances; and

classifying the unlabeled query data set via the trained version of the supervised anomaly detection classifier.

2. The method of claim 1 wherein the method further comprises, subsequently to training the supervised anomaly detection classifier:

classifying the unlabeled training data set used to train the unsupervised anomaly detection classifier via the trained version of the supervised anomaly detection classifier, the classifying comprising, for each unlabeled data instance in the unlabeled training data set:

providing the unlabeled data instance as a query input to the trained version of the supervised anomaly detection classifier; and

generating, via the trained version of the supervised anomaly detection classifier, a predicted classification and a confidence level for the unlabeled data instance;

updating the labeled training data set based on the confidence levels generated as a result of classifying the unlabeled training data set via the trained version of the supervised anomaly detection classifier; and

re-training the trained version of the supervised anomaly detection classifier using the updated labeled training data set.

3. The method of claim 2 further comprising repeating the classifying, the updating, and the re-training recited in claim 2 until a predefined criterion is met or a predefined iteration limit is reached.

4. The method of claim 1 wherein constructing the labeled training data set comprises, for each unlabeled data instance in the first and second subsets, adding the unlabeled data instance's anomaly score as an additional feature of the labeled data instance created from the unlabeled data instance.

5. The method of claim 4 wherein the method further comprises, prior to classifying the unlabeled query data set via the trained version of the supervised anomaly detection classifier:

classifying the unlabeled query data set via the trained version of the unsupervised anomaly detection classifier, the classifying resulting in anomaly scores for said another plurality of unlabeled data instances; and

for each unlabeled data instance in the unlabeled query data set, adding the unlabeled data instance's anomaly score as an additional feature to the unlabeled data instance.

6. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code causing the computer system to execute a method comprising:

training an unsupervised anomaly detection classifier using an unlabeled training data set comprising a plurality of unlabeled data instances, the training resulting in a trained version of the unsupervised anomaly detection classifier;

classifying the unlabeled training data set used to train the unsupervised anomaly detection classifier via the trained version of the unsupervised anomaly detection classifier, the classifying comprising, for each unlabeled data instance in the unlabeled training data set:

providing the unlabeled data instance as a query input to the trained version of the unsupervised anomaly detection classifier; and

generating, via the trained version of the unsupervised anomaly detection classifier, an anomaly score for the unlabeled data instance;

constructing a labeled training data set obtained from the trained version of the unsupervised anomaly detection classifier that is derived from a first subset of unlabeled data instances in the unlabeled training data set whose anomaly scores are below a first threshold and a second subset of unlabeled data instances in the unlabeled training data set whose anomaly scores are above a second threshold, the constructing comprising, for each unlabeled data instance in the unlabeled training data set:

upon determining that the anomaly score for the unlabeled data instance is below the first threshold, adding the unlabeled data instance as a new labeled data instance to the labeled training data set with a class label indicating that the new labeled data instance is normal; and

upon determining that the anomaly score for the unlabeled data instance is above the second threshold, adding the unlabeled data instance as a new labeled data instance to the labeled training data set with a class label indicating that the new labeled data instance is anomalous;

training a supervised anomaly detection classifier using the labeled training data set obtained from the trained version of the unsupervised anomaly detection classifier, the training resulting in a trained version of the supervised anomaly detection classifier;

receiving an unlabeled query data set comprising another plurality of unlabeled data instances; and

classifying the unlabeled query data set via the trained version of the supervised anomaly detection classifier.

7. The non-transitory computer readable storage medium of claim 6 wherein the method further comprises, subsequently to training the supervised anomaly detection classifier:

classifying the unlabeled training data set used to train the unsupervised anomaly detection classifier via the trained version of the supervised anomaly detection classifier, the classifying comprising, for each unlabeled data instance in the unlabeled training data set:

providing the unlabeled data instance as a query input to the trained version of the supervised anomaly detection classifier; and

generating, via the trained version of the supervised anomaly detection classifier, a predicted classification and a confidence level for the unlabeled data instance;

updating the labeled training data set based on the confidence levels generated as a result of classifying the unlabeled training data set via the trained version of the supervised anomaly detection classifier; and

re-training the trained version of the supervised anomaly detection classifier using the updated labeled training data set.

8. The non-transitory computer readable storage medium of claim 7 wherein the method further comprises repeating the classifying, the updating, and the re-training recited in claim 7 until a predefined criterion is met or a predefined iteration limit is reached.

9. The non-transitory computer readable storage medium of claim 6 wherein constructing the labeled training data set comprises, for each unlabeled data instance in the first and second subsets, adding the unlabeled data instance's anomaly score as an additional feature of the labeled data instance created from the unlabeled data instance.

10. The non-transitory computer readable storage medium of claim 9 wherein the method further comprises, prior to classifying the unlabeled query data set via the trained version of the supervised anomaly detection classifier:

classifying the unlabeled query data set via the trained version of the unsupervised anomaly detection classifier, the classifying resulting in anomaly scores for said another plurality of unlabeled data instances; and

for each unlabeled data instance in the unlabeled query data set, adding the unlabeled data instance's anomaly score as an additional feature to the unlabeled data instance.

11. A computer system comprising:

a processor; and

a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:

train an unsupervised anomaly detection classifier using an unlabeled training data set comprising a plurality of unlabeled data instances, the training resulting in a trained version of the unsupervised anomaly detection classifier;

classify the unlabeled training data set used to train the unsupervised anomaly detection classifier via the trained version of the unsupervised anomaly detection classifier, the classifying comprising, for each unlabeled data instance in the unlabeled training data set:

providing the unlabeled data instance as a query input to the trained version of the unsupervised anomaly detection classifier; and

generating, via the trained version of the unsupervised anomaly detection classifier, an anomaly score for the unlabeled data instance;

construct a labeled training data set obtained from the trained version of the unsupervised anomaly detection classifier that is derived from a first subset of unlabeled data instances in the unlabeled training data set whose anomaly scores are below a first threshold and a second subset of unlabeled data instances in the unlabeled training data set whose anomaly scores are above a second threshold, the constructing comprising, for each unlabeled data instance in the unlabeled training data set:

upon determining that the anomaly score for the unlabeled data instance is below the first threshold, adding the unlabeled data instance as a new labeled data instance to the labeled training data set with a class label indicating that the new labeled data instance is normal; and

upon determining that the anomaly score for the unlabeled data instance is above the second threshold, adding the unlabeled data instance as a new labeled data instance to the labeled training data set with a class label indicating that the new labeled data instance is anomalous;

train a supervised anomaly detection classifier using the labeled training data set obtained from the trained version of the unsupervised anomaly detection classifier, the training resulting in a trained version of the supervised anomaly detection classifier;

receive an unlabeled query data set comprising another plurality of unlabeled data instances; and

classify the unlabeled query data set via the trained version of the supervised anomaly detection classifier.

12. The computer system of claim 11 wherein the program code further causes the processor to, subsequently to training the supervised anomaly detection classifier:

classify the unlabeled training data set used to train the unsupervised anomaly detection classifier via the trained version of the supervised anomaly detection classifier, the classifying comprising, for each unlabeled data instance in the unlabeled training data set:

providing the unlabeled data instance as a query input to the trained version of the supervised anomaly detection classifier; and

generating, via the trained version of the supervised anomaly detection classifier, a predicted classification and a confidence level for the unlabeled data instance;

update the labeled training data set based on the confidence levels generated as a result of classifying the unlabeled training data set via the trained version of the supervised anomaly detection classifier; and

re-train the trained version of the supervised anomaly detection classifier using the updated labeled training data set.

13. The computer system of claim 12 wherein the classifying, the updating, and the re-training recited in claim 12 are repeated until a predefined criterion is met or a predefined iteration limit is reached.

14. The computer system of claim 11 wherein the program code that causes the processor to construct the labeled training data set comprises program code that causes the processor to, for each unlabeled data instance in the first and second subsets:

add the unlabeled data instance's anomaly score as an additional feature of the labeled data instance created from the unlabeled data instance.

15. The computer system of claim 14 wherein the program code further causes the processor to, prior to classifying the unlabeled query data set via the trained version of the supervised anomaly detection classifier:

classify the unlabeled query data set via the trained version of the unsupervised anomaly detection classifier, the classifying resulting in anomaly scores for said another plurality of unlabeled data instances; and

for each unlabeled data instance in the unlabeled query data set, add the unlabeled data instance's anomaly score as an additional feature to the unlabeled data instance.