Ensemble of clustered dual-stage attention-based recurrent neural networks for multivariate time series prediction
A method for multivariate time series prediction is provided. Each time series from among a batch of multiple driving time series and a target time series is decomposed into a raw component, a shape component, and a trend component. For each decomposed component, select a driving time series relevant thereto from the batch and obtain hidden features of the selected driving time series, by applying the batch to an input attention-based encoder of an Ensemble of Clustered dual-stage attention-based Recurrent Neural Networks (EC-DARNNS). Automatically cluster the hidden features in a hidden space using a temporal attention-based decoder of the EC-DARNNS. Each Clustered dual-stage attention-based RNN in the Ensemble is dedicated and applied to a respective one of the decomposed components. Predict a respective value of one or more future time steps for the target series based on respective prediction outputs for each of the decomposed components by the EC-DARNNS.
1. A computer-implemented method for multivariate time series prediction, comprising:
decomposing each time series from among a batch of multiple driving time series and a target time series into a raw component, a shape component, and a trend component, the shape component including an original time series segment minus values of each time series at T time step, and the trend component including a value difference of two consecutive time steps;
for each of the decomposed components, selecting a driving time series relevant thereto from the batch and obtaining hidden features of the selected driving time series, by applying the batch to an input attention-based encoder of an Ensemble of Clustered dual-stage attention-based Recurrent Neural Networks (EC-DARNNS);
automatically clustering, by a hardware processor, the hidden features in a hidden space using a temporal attention-based decoder of the EC-DARNNS, each of the Clustered dual-stage attention-based Recurrent Neural Networks (RNNs) in the Ensemble being dedicated and applied to a respective one of the decomposed components; and
predicting a respective value of one or more future time steps for the target time series based on respective prediction outputs for each of the decomposed components by the EC-DARNNS.
2. The computer-implemented method of claim 1 , wherein said automatically clustering step automatically clusters the hidden features in the hidden space based upon a Gaussian Mixture Model (GMM).
3. The computer-implemented method of claim 2 , wherein the Gaussian Mixture Model is used for hidden feature regularization.
4. The computer-implemented method of claim 3 , wherein the hidden features are regularized with respect to a GMM loss implemented as a regularization term.
5. The computer-implemented method of claim 1 , wherein the batch of multiple driving time series represents historical data.
6. The computer-implemented method of claim 1 , wherein the input attention-based encoder comprises a plurality of Long-Short-Term Memories (LTSMs).
7. The computer-implemented method of claim 6 , wherein the input attention-based encoder further comprises one or more input attention layers followed h one or more softmax layers.
8. The computer-implemented method of claim 7 , wherein the plurality of LSTMs precede the one or more input attention layers.
9. The computer-implemented method of claim 1 , wherein the temporal attention-based decoder comprises a plurality of Long-Short-Term Memories (LTSMs).
10. The computer-implemented method of claim 9 , wherein the temporal attention-based decoder further comprises one or more temporal attention layers followed by one or more softmax layers.
11. The computer-implemented method of claim 10 , wherein the plurality of LSTMs precede the one or more temporal attention layers.
12. The computer-implemented method of claim 1 , wherein the target time series comprises time series data of at least one hardware sensor for anomaly detection of a hardware system.
13. The computer-implemented method of claim 12 , wherein the hardware system is controlled responsive to the respective value of the one or more future time steps.
14. A computer program product for multivariate time series prediction, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
decomposing each time series from among a batch of multiple driving time series and a target time series into a raw component, a shape component, and a trend component, the shape component including an original time series segment minus values of each time series at T time step, and the trend component including a value difference of two consecutive time steps;
for each of the decomposed components, selecting a driving time series relevant thereto from the batch and obtaining hidden features of the selected driving time series, by applying the batch to an input attention-based encoder of an Ensemble of Clustered dual-stage attention-based Recurrent Neural Networks (EC-DARNNS);
automatically clustering the hidden features in a hidden space using a temporal attention-based decoder of the EC-DARNNS, each of the Clustered dual-stage attention-based Recurrent Neural Networks (RNNs) in the Ensemble being dedicated and applied to a respective one of the decomposed components; and
predicting a respective value of one or more future time steps for the target time series based on respective prediction outputs for each of the decomposed components by the EC-DARNNS.
15. The computer program product of claim 14 , wherein said automatically clustering step automatically clusters the hidden features in the hidden space based upon a Gaussian Mixture Model (GMM).
16. The computer program product of claim 15 , wherein the Gaussian Mixture Model is used for hidden feature regularization.
17. The computer program product of claim 16 , wherein the hidden features are regularized with respect to a GMM loss implemented as a regularization term.
18. The computer program product of claim 14 , wherein the batch of multiple driving time series represents historical data.
19. The computer program product of claim 14 , wherein the input attention-based encoder comprises a plurality of Long-Short-Term Memories (LTSMs).
20. A computer processing system for multivariate time series prediction, comprising:
a memory for storing program code; and
a processor operatively coupled to the memory for executing the program code to:
decompose each time series from among a batch of multiple driving time series and a target time series into a raw component, a shape component, and a trend component, the shape component including an original time series segment minus values of each time series at T time step, and the trend component including a value difference of two consecutive time steps;
for each of the decomposed components, select a driving time series relevant thereto from the batch and obtain hidden features of the selected driving time series, by applying the batch to an input attention-based encoder of an Ensemble of Clustered dual-stage attention-based Recurrent Neural Networks (EC-DARNNS);
automatically cluster the hidden features in a hidden space using a temporal attention-based decoder of the EC-DARNNS, each of the Clustered dual-stage attention-based Recurrent Neural Networks (RNNs) in the Ensemble being dedicated and applied to a respective one of the decomposed components; and
predict a respective value of one or more future time steps for the target time series based on respective prediction outputs for each of the decomposed components by the EC-DARNNS.