Learning hyper-parameter scaling models for unsupervised anomaly detection
A computer sorts empirical validation scores of validated training scenarios of an anomaly detector. Each training scenario has a dataset to train an instance of the anomaly detector that is configured with values for hyperparameters. Each dataset has values for metafeatures. For each predefined ranking percentage, a subset of best training scenarios is selected that consists of the ranking percentage of validated training scenarios having the highest empirical validation scores. Linear optimizers train to infer a value for a hyperparameter. Into many distinct unvalidated training scenarios, a scenario is generated that has metafeatures values and hyperparameters values that contains the value inferred for that hyperparameter by a linear optimizer. For each unvalidated training scenario, a validation score is inferred. A best linear optimizer is selected having a highest combined inferred validation score. For a new dataset, the best linear optimizer infers a value of that hyperparameter.
1 . A method comprising:
sorting respective empirical validation scores of a plurality of validated training scenarios of an anomaly detector, wherein:
each validated training scenario has a respective training dataset to train a respective instance of the anomaly detector that is respectively configured with a respective plurality of values for a plurality of hyperparameters of the anomaly detector, and
said respective training dataset has a respective set of values for a set of metafeatures;
for each ranking percentage of a plurality of predefined distinct ranking percentages:
a) selecting a subset of best training scenarios that consists of the ranking percentage of the plurality of validated training scenarios of the anomaly detector having the highest empirical validation scores;
b) training a respective linear optimizer that infers a respective value for a particular hyperparameter of the plurality of hyperparameters of the anomaly detector, wherein the training is based on the sets of values of the set of metafeatures of the training datasets of the subset of best training scenarios and the empirical validation scores of the subset of best training scenarios,
c) for each set of metafeatures values in a plurality of unvalidated sets of values of the set of metafeatures, wherein the set of metafeatures values does not correspond to an actual dataset:
i) inferring, by the linear optimizer, a particular value for the particular hyperparameter of the plurality of hyperparameters of the anomaly detector;
ii) generating, into a plurality of distinct unvalidated training scenarios, a distinct unvalidated training scenario of the anomaly detector that has: the set of metafeatures values and a plurality of values for the plurality of hyperparameters of the anomaly detector that contains the particular value for the particular hyperparameter;
inferring, for each unvalidated training scenario of the plurality of distinct unvalidated training scenarios of the anomaly detector, a respective inferred validation score;
selecting a best linear optimizer of a ranking percentage of the plurality of predefined distinct ranking percentages having a highest combined inferred validation score based on said inferred validation scores for the plurality of distinct unvalidated training scenarios of the anomaly detector;
inferring, by the best linear optimizer of the ranking percentage of the plurality of predefined distinct ranking percentages, for a new dataset, an inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector; and
detecting, by a new instance of the anomaly detector that is based on the inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector, an anomaly;
wherein the method is performed by one or more computers.
2 . The method of claim 1 further comprising training a metamodel to infer a validation score of a training scenario of the anomaly detector, wherein the metamodel performs said inferring the validation scores, and the training the metamodel is based on:
the empirical validation scores of the plurality of validated training scenarios of the anomaly detector,
the sets of values of the set of metafeatures of the training datasets of the plurality of validated training scenarios, and
the pluralities of values of the plurality of hyperparameters of the anomaly detector.
3 . The method of claim 2 wherein one selected from the group consisting of:
said training the metamodel occurs after said training the linear optimizers, and
said training the metamodel and said training the linear optimizers concurrently occur.
4 . The method of claim 2 further comprising:
inferring, by a second optimizer for a second particular hyperparameter of the plurality of hyperparameters of the anomaly detector, a particular value for the second particular hyperparameter;
generating, into a second plurality of distinct unvalidated training scenarios, a distinct unvalidated training scenario of the anomaly detector that has a plurality of values for the plurality of hyperparameters of the anomaly detector that contains the particular value for the second particular hyperparameter;
inferring, by the metamodel, a respective inferred validation score for each unvalidated training scenario of the second plurality of distinct unvalidated training scenarios of the anomaly detector.
5 . The method of claim 4 wherein said second optimizer is one selected from the group consisting of a linear regression and a logistic regression.
6 . The method of claim 2 wherein:
the plurality of validated training scenarios of the anomaly detector contains:
a first training scenario that has a first training dataset that consists of representations of instances of a first kind of object, and
a second training scenario that has a second training dataset that consists of representations of instances of a second kind of object;
the first kind of object and the second kind of object are distinct kinds of objects selected from the group consisting of: a network packet, a communicated message, a log entry, a logic statement, a semi-structured document, a photograph, a database record, a logical graph, and a parse tree.
7 . The method of claim 2 wherein:
said anomaly detector is a first anomaly detector;
said plurality of hyperparameters of the first anomaly detector is disjoint from a plurality of hyperparameters of a second anomaly detector;
the method further comprises:
inferring, by a linear optimizer for a particular hyperparameter of the plurality of hyperparameters of the second anomaly detector, a particular value for the particular hyperparameter of the plurality of hyperparameters of the second anomaly detector;
inferring, by a second metamodel, a respective inferred validation score for an unvalidated training scenario of the second anomaly detector that has the particular value for the particular hyperparameter of the plurality of hyperparameters of the second anomaly detector.
8 . The method of claim 7 further comprising combining, in an ensemble, an instance of the first anomaly detector and an instance of the second anomaly detector.
9 . The method of claim 2 wherein the metamodel does not comprise an artificial neural network.
10 . The method of claim 1 wherein at least one selected from the group consisting of:
said set of metafeatures consists only of a neutral metafeature,
said set of metafeatures contains a dataset cardinality metafeature,
said best linear optimizer linearly scales the particular hyperparameter based on only one metafeature, and
said best linear optimizer monotonically scales the particular hyperparameter.
11 . The method of claim 10 further comprising:
before said inferring the inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector, tuning the particular hyperparameter based on a subset of the new dataset to provide an initial value of the particular hyperparameter;
training, based on the entire new dataset, an instance of the anomaly detector that is based on the inferred value of the particular hyperparameter.
12 . The method of claim 1 wherein the training datasets of the plurality of validated training scenarios of the anomaly detector contain at least one selected from the group consisting of:
two training datasets that have distinct respective dimensionalities, and
two training datasets that have respective distinct pluralities of metafeatures that include said set of metafeatures.
13 . The method of claim 1 wherein:
a step is based on an unlabeled training dataset;
the step is at least one selected from the group consisting of:
said inferring the inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector, and
said detecting the anomaly by said new instance of the anomaly detector that is based on the inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector.
14 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
sorting respective empirical validation scores of a plurality of validated training scenarios of an anomaly detector, wherein:
each validated training scenario has a respective training dataset to train a respective instance of the anomaly detector that is respectively configured with a respective plurality of values for a plurality of hyperparameters of the anomaly detector, and
said respective training dataset has a respective set of values for a set of metafeatures;
for each ranking percentage of a plurality of predefined distinct ranking percentages:
a) selecting a subset of best training scenarios that consists of the ranking percentage of the plurality of validated training scenarios of the anomaly detector having the highest empirical validation scores;
b) training a respective linear optimizer that infers a respective value for a particular hyperparameter of the plurality of hyperparameters of the anomaly detector, wherein the training is based on the sets of values of the set of metafeatures of the training datasets of the subset of best training scenarios and the empirical validation scores of the subset of best training scenarios,
c) for each set of metafeatures values in a plurality of unvalidated sets of values of the set of metafeatures, wherein the set of metafeatures values does not correspond to an actual dataset:
i) inferring, by the linear optimizer, a particular value for the particular hyperparameter of the plurality of hyperparameters of the anomaly detector;
ii) generating, into a plurality of distinct unvalidated training scenarios, a distinct unvalidated training scenario of the anomaly detector that has: the set of metafeatures values and a plurality of values for the plurality of hyperparameters of the anomaly detector that contains the particular value for the particular hyperparameter;
inferring, for each unvalidated training scenario of the plurality of distinct unvalidated training scenarios of the anomaly detector, a respective inferred validation score;
selecting a best linear optimizer of a ranking percentage of the plurality of predefined distinct ranking percentages having a highest combined inferred validation score based on said inferred validation scores for the plurality of distinct unvalidated training scenarios of the anomaly detector;
inferring, by the best linear optimizer of the ranking percentage of the plurality of predefined distinct ranking percentages, for a new dataset, an inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector; and
detecting, by a new instance of the anomaly detector that is based on the inferred value of the particular hyperparameter of the plurality of hyperparameters of the anomaly detector, an anomaly.
15 . The one or more non-transitory computer-readable media of claim 14 wherein the instructions further cause training a metamodel to infer a validation score of a training scenario of the anomaly detector, wherein the metamodel performs said inferring the validation scores, and the training the metamodel is based on:
the empirical validation scores of the plurality of validated training scenarios of the anomaly detector,
the sets of values of the set of metafeatures of the training datasets of the plurality of validated training scenarios, and
the pluralities of values of the plurality of hyperparameters of the anomaly detector.
16 . The one or more non-transitory computer-readable media of claim 15 wherein one selected from the group consisting of:
said training the metamodel occurs after said training the linear optimizers, and
said training the metamodel and said training the linear optimizers concurrently occur.
17 . The one or more non-transitory computer-readable media of claim 15 wherein the instructions further cause:
inferring, by a second optimizer for a second particular hyperparameter of the plurality of hyperparameters of the anomaly detector, a particular value for the second particular hyperparameter;
generating, into a second plurality of distinct unvalidated training scenarios, a distinct unvalidated training scenario of the anomaly detector that has a plurality of values for the plurality of hyperparameters of the anomaly detector that contains the particular value for the second particular hyperparameter;
inferring, by the metamodel, a respective inferred validation score for each unvalidated training scenario of the second plurality of distinct unvalidated training scenarios of the anomaly detector.
18 . The one or more non-transitory computer-readable media of claim 15 wherein:
the plurality of validated training scenarios of the anomaly detector contains:
a first training scenario that has a first training dataset that consists of representations of instances of a first kind of object, and
a second training scenario that has a second training dataset that consists of representations of instances of a second kind of object;
the first kind of object and the second kind of object are distinct kinds of objects selected from the group consisting of: a network packet, a communicated message, a log entry, a logic statement, a semi-structured document, a photograph, a database record, a logical graph, and a parse tree.
19 . The one or more non-transitory computer-readable media of claim 15 wherein:
said anomaly detector is a first anomaly detector;
said plurality of hyperparameters of the first anomaly detector is disjoint from a plurality of hyperparameters of a second anomaly detector;
the instructions further cause:
inferring, by a linear optimizer for a particular hyperparameter of the plurality of hyperparameters of the second anomaly detector, a particular value for the particular hyperparameter of the plurality of hyperparameters of the second anomaly detector;
inferring, by a second metamodel, a respective inferred validation score for an unvalidated training scenario of the second anomaly detector that has the particular value for the particular hyperparameter of the plurality of hyperparameters of the second anomaly detector.
20 . The one or more non-transitory computer-readable media of claim 14 wherein the training datasets of the plurality of validated training scenarios of the anomaly detector contain at least one selected from the group consisting of:
two training datasets that have distinct respective dimensionalities, and
two training datasets that have respective distinct pluralities of metafeatures that include said set of metafeatures.