System, method, and computer program product for multi-domain ensemble learning based on multivariate time sequence data
Systems, methods, and computer program products for multi-domain ensemble learning based on multivariate time sequence data are provided. A method may include receiving multivariate sequence data. At least a portion of the multivariate sequence data may be inputted into a plurality of anomaly detection models to generate a plurality of scores. The multivariate sequence data may be combined with the plurality of scores to generate combined intermediate data. The combined intermediate data may be inputted into a combined ensemble model to generate an output score. In response to determining that the output score satisfies a threshold, at least one of an alert may be communicated to a user device, the multivariate sequence data may be inputted into the feature-domain ensemble model to generate a feature importance vector, or at least one of a model-domain, a time-domain, a feature-domain, or the combined ensemble model may be updated.
1. A computer-implemented method, comprising:
receiving, with at least one processor, multivariate sequence data comprising a plurality of vectors, each vector of the plurality of vectors comprising elements based on a time sequence for a respective variable of a plurality of variables;
generating, with the at least one processor, a plurality of scores comprising a respective score for each anomaly detection model based on inputting at least a portion of the multivariate sequence data into each anomaly detection model of a plurality of anomaly detection models;
combining, with the at least one processor, the multivariate sequence data with the plurality of scores to generate combined intermediate data;
generating, with the at least one processor, an output score based on inputting the combined intermediate data into a time-series sequences ensemble model (TSEM), the TSEM based on a model-domain ensemble model, a time-domain ensemble model, and a feature-domain ensemble model;
determining, with the at least one processor, that the output score generated based on the combined intermediate data satisfies a threshold; and
in response to determining that the output score generated based on the combined intermediate data satisfies the threshold:
updating, with the at least one processor, parameters of at least one of the TSEM or the feature-domain ensemble model, wherein updating the parameters of at least one of the TSEM or the feature-domain ensemble model comprises, for every time step less than a maximum time step:
determining, with the at least one processor, a first variable based on the combined intermediate data and a current time step version of the TSEM;
determining, with the at least one processor, a second variable based on a ratio of labels to the first variable;
backpropagating, with the at least one processor, the feature-domain ensemble model based on a transpose of the combined intermediate data and the second variable; and
determining, with the at least one processor, a next time step version of the TSEM based on backpropagation of the feature-domain ensemble model and the current time step version of the TSEM.
2. The method of claim 1 , wherein each anomaly detection model of the plurality of anomaly detection models comprises at least one of a classifier model or a score generation model.
3. The method of claim 1 , wherein each anomaly detection model of the plurality of anomaly detection models comprises at least one of a Bayesian model, a Kullback-Leibler importance estimation procedure (KLIEP) model, a ChangeFinder model, or a cumulative sum (CUSUM) model.
4. The method of claim 1 , further comprising:
determining, with at least one processor, whether to label or communicate the at least the portion of the multivariate sequence data based on the output score; and
in response to determining whether to label or communicate the at least the portion of the multivariate sequence data, one of:
labeling, with at least one processor, the at least the portion of the multivariate sequence data based on the output score; or
communicating, with at least one processor, the at least the portion of the multivariate sequence data and the output score to the user device.
5. The method of claim 1 , wherein updating the parameters of at least one of the TSEM or the feature-domain ensemble model comprises:
initializing, with at least one processor, the TSEM.
6. The method of claim 5 , wherein updating the parameters of at least one of the TSEM or the feature-domain ensemble model further comprises:
for every time step less than the maximum time step:
determining, with at least one processor, a third variable based on a first Khatri-Rao product based on a current time step version of the time-domain ensemble model and a current time step version of the feature-domain ensemble model;
determining, with at least one processor, a current time step version of the model-domain ensemble model based on the third variable, the model-domain ensemble model, and a tensor unfolding in mode 0 of the TSEM;
determining, with at least one processor, a fourth variable based on a second Khatri-Rao product based on the current time step version of the model-domain ensemble model and the current time step version of the feature-domain ensemble model;
determining, with at least one processor, an updated current time step version of the time-domain ensemble model based on the fourth variable, the time-domain ensemble model, and a tensor unfolding in mode 1 of the TSEM;
determining, with at least one processor, a fifth variable based on a third Khatri-Rao product based on the current time step version of the model-domain ensemble model and the updated current time step version of the time-domain ensemble model; and
determining, with at least one processor, an updated current time step version of the feature-domain ensemble model based on the fifth variable, the feature-domain ensemble model, and a tensor unfolding in mode 2 of the TSEM.
7. The method of claim 1 , wherein a loss function of the TSEM is based on the model-domain ensemble model, the time-domain ensemble model, and the feature-domain ensemble model.
8. The method of claim 7 , wherein the loss function of the TSEM is based on the following equation:
L
(
M
,
T
,
F
)
=
1
2
❘
"\[LeftBracketingBar]"
H
-
MTF
❘
"\[RightBracketingBar]"
2
+
λ
1
2
(
❘
"\[LeftBracketingBar]"
M
❘
"\[RightBracketingBar]"
2
+
❘
"\[LeftBracketingBar]"
T
❘
"\[RightBracketingBar]"
2
+
❘
"\[LeftBracketingBar]"
F
❘
"\[RightBracketingBar]"
2
)
+
λ
2
2
❘
"\[LeftBracketingBar]"
l
-
ZF
❘
"\[RightBracketingBar]"
2
wherein M is the model-domain ensemble model, T is the time-domain ensemble model, F is the feature-domain ensemble model, H is the TSEM, I is the feature importance vector, Z is the multivariate sequence data, λ 1 is a first coefficient, and λ 2 is a second coefficient.
9. A system, comprising:
at least one processor; and
at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, direct the at least one processor to:
receive multivariate sequence data comprising a plurality of vectors, each vector of the plurality of vectors comprising elements based on a time sequence for a respective variable of a plurality of variables;
generate a plurality of scores comprising a respective score for each anomaly detection model based on inputting at least a portion of the multivariate sequence data into each anomaly detection model of a plurality of anomaly detection models;
combine the multivariate sequence data with the plurality of scores to generate combined intermediate data;
generate an output score based on inputting the combined intermediate data into a time-series sequences ensemble model (TSEM), the TSEM based on a model-domain ensemble model, a time-domain ensemble model, and a feature-domain ensemble model;
determine that the output score generated based on the combined intermediate data satisfies a threshold; and
in response to determining that the output score generated based on the combined intermediate data satisfies the threshold:
update parameters of at least one of the TSEM or the feature-domain ensemble model, wherein, when updating the parameters of at least one of the TSEM or the feature-domain ensemble model, the one or more instructions further direct the at least one processor to, for every time step less than a maximum time step:
determine a first variable based on the combined intermediate data and a current time step version of the TSEM;
determine a second variable based on a ratio of labels to the first variable;
backpropagate the feature-domain ensemble model based on a transpose of the combined intermediate data and the second variable; and
determine a next time step version of the TSEM based on backpropagation of the feature-domain ensemble model and the current time step version of the TSEM.
10. The system of claim 9 , wherein each anomaly detection model of the plurality of anomaly detection models comprises at least one of a classifier model or a score generation model.
11. The system of claim 9 , wherein each anomaly detection model of the plurality of anomaly detection models comprises at least one of a Bayesian model, a Kullback-Leibler importance estimation procedure (KLIEP) model, a ChangeFinder model, or a cumulative sum (CUSUM) model.
12. The system of claim 9 , wherein the one or more instructions further direct the at least one processor to:
determine whether to label or communicate the at least the portion of the multivariate sequence data based on the output score; and
in response to determining whether to label or communicate the at least the portion of the multivariate sequence data, one of:
label the at least the portion of the multivariate sequence data based on the output score; or
communicate the at least the portion of the multivariate sequence data and the output score to the user device.
13. The system of claim 9 , wherein, when updating the parameters of at least one of the TSEM or the feature-domain ensemble model, the one or more instructions further direct the at least one processor to:
initialize the TSEM.
14. The system of claim 13 , wherein, when updating the parameters of at least one of the TSEM or the feature-domain ensemble model, the one or more instructions further direct the at least one processor to:
for every time step less than the maximum time step:
determine a third variable based on a first Khatri-Rao product based on a current time step version of the time-domain ensemble model and a current time step version of the feature-domain ensemble model;
determine a current time step version of the model-domain ensemble model based on the third variable, the model-domain ensemble model, and a tensor unfolding in mode 0 of the TSEM;
determine a fourth variable based on a second Khatri-Rao product based on the current time step version of the model-domain ensemble model and the current time step version of the feature-domain ensemble model;
determine an updated current time step version of the time-domain ensemble model based on the fourth variable, the time-domain ensemble model, and a tensor unfolding in mode 1 of the TSEM;
determine a fifth variable based on a third Khatri-Rao product based on the current time step version of the model-domain ensemble model and the updated current time step version of the time-domain ensemble model; and
determine an updated current time step version of the feature-domain ensemble model based on the fifth variable, the feature-domain ensemble model, and a tensor unfolding in mode 2 of the TSEM.
15. The system of claim 9 , wherein a loss function of the TSEM is based on the model-domain ensemble model, the time-domain ensemble model, and the feature-domain ensemble model.
16. The system of claim 15 , wherein the loss function of the TSEM is based on the following equation:
L
(
M
,
T
,
F
)
=
1
2
❘
"\[LeftBracketingBar]"
H
-
MTF
❘
"\[RightBracketingBar]"
2
+
λ
1
2
(
❘
"\[LeftBracketingBar]"
M
❘
"\[RightBracketingBar]"
2
+
❘
"\[LeftBracketingBar]"
T
❘
"\[RightBracketingBar]"
2
+
❘
"\[LeftBracketingBar]"
F
❘
"\[RightBracketingBar]"
2
)
+
λ
2
2
❘
"\[LeftBracketingBar]"
l
-
ZF
❘
"\[RightBracketingBar]"
2
wherein M is the model-domain ensemble model, T is the time-domain ensemble model, F is the feature-domain ensemble model, H is the TSEM, I is the feature importance vector, Z is the multivariate sequence data, λ 1 is a first coefficient, and λ 2 is a second coefficient.
17. A computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to:
receive multivariate sequence data comprising a plurality of vectors, each vector of the plurality of vectors comprising elements based on a time sequence for a respective variable of a plurality of variables;
generate a plurality of scores comprising a respective score for each anomaly detection model based on inputting at least a portion of the multivariate sequence data into each anomaly detection model of a plurality of anomaly detection models;
combine the multivariate sequence data with the plurality of scores to generate combined intermediate data;
generate an output score based on inputting the combined intermediate data into a time-series sequences ensemble model (TSEM), the TSEM based on a model-domain ensemble model, a time-domain ensemble model, and a feature-domain ensemble model;
determine that the output score generated based on the combined intermediate data satisfies a threshold; and
in response to determining that the output score generated based on the combined intermediate data satisfies the threshold:
update parameters of at least one of the TSEM or the feature-domain ensemble model, wherein, when updating the parameters of at least one of the TSEM or the feature-domain ensemble model, the one or more instructions further cause the at least one processor to, for every time step less than a maximum time step:
determine a first variable based on the combined intermediate data and a current time step version of the TSEM;
determine a second variable based on a ratio of labels to the first variable;
backpropagate the feature-domain ensemble model based on a transpose of the combined intermediate data and the second variable; and
determine a next time step version of the TSEM based on backpropagation of the feature-domain ensemble model and the current time step version of the TSEM.
18. The computer program product of claim 17 , wherein the one or more instructions further cause the at least one processor to:
determine whether to label or communicate the at least the portion of the multivariate sequence data based on the output score; and
in response to determining whether to label or communicate the at least the portion of the multivariate sequence data, one of:
label the at least the portion of the multivariate sequence data based on the output score; or
communicate the at least the portion of the multivariate sequence data and the output score to the user device.
19. The computer program product of claim 17 , wherein, when updating the parameters of at least one of the TSEM or the feature-domain ensemble model, the one or more instructions further cause the at least one processor to:
initialize the TSEM.
20. The computer program product of claim 19 , wherein, when updating the parameters of at least one of the TSEM or the feature-domain ensemble model, the one or more instructions further cause the at least one processor to:
for every time step less than the maximum time step:
determine a third variable based on a first Khatri-Rao product based on a current time step version of the time-domain ensemble model and a current time step version of the feature-domain ensemble model;
determine a current time step version of the model-domain ensemble model based on the third variable, the model-domain ensemble model, and a tensor unfolding in mode 0 of the TSEM;
determine a fourth variable based on a second Khatri-Rao product based on the current time step version of the model-domain ensemble model and the current time step version of the feature-domain ensemble model;
determine an updated current time step version of the time-domain ensemble model based on the fourth variable, the time-domain ensemble model, and a tensor unfolding in mode 1 of the TSEM;
determine a fifth variable based on a third Khatri-Rao product based on the current time step version of the model-domain ensemble model and the updated current time step version of the time-domain ensemble model; and
determine an updated current time step version of the feature-domain ensemble model based on the fifth variable, the feature-domain ensemble model, and a tensor unfolding in mode 2 of the TSEM.