Tensorized LSTM with adaptive shared memory for learning trends in multivariate time series
A method for executing a multi-task deep learning model for learning trends in multivariate time series is presented. The method includes collecting multi-variate time series data from a plurality of sensors, jointly learning both local and global contextual features for predicting a trend of the multivariate time series by employing a tensorized long short-term memory (LSTM) with adaptive shared memory (TLASM) to learn historical dependency of historical trends, and employing a multi-task one-dimensional convolutional neural network (1dCNN) to extract salient features from local raw time series data to model a short-term dependency between local time series data and subsequent trends.
1. A computer-implemented method executed on a processor for executing a multi-task deep learning model for learning trends in multivariate time series, the method comprising:
collecting multi-variate time series data from a plurality of sensors;
jointly learning both local and global contextual features for predicting a trend of the multivariate time series by employing a tensorized long short-term memory (LSTM) with adaptive shared memory (TLASM) to learn historical dependency of historical trends, the adaptive shared memory including multiple layers of sub-networks, wherein the sub-networks are connected in parallel with each other, a first layer of each sub-network including multiple independent LSTMs followed by multiple layers of multilayer perceptrons (MLPs), and a last layer of the adaptive shared memory including task-specific MLPs, wherein all the independent LSTMs and the sub-networks are shared by all prediction tasks, and a connection between the sub-networks is a weighted average with an attention mechanism;
employing a multi-task one-dimensional convolutional neural network (1dCNN) to extract salient features from local raw time series data to model a short-term dependency between local time series data and subsequent trends, wherein each of the independent LSTMs directly communicates with each MLP of a first layer of MLPs of the multiple layers of MLPs; and
concatenating outputs of the TLASM and the multi-task one-dimensional convolutional neural network and the concatenated output are fed into a task-specific sub-network, wherein the task-specific sub-network generates trend prediction for the multivariate time series data and wherein the task-specific subnetwork includes an MLP layer.
2. The method of claim 1 , wherein the adaptive shared memory models task relatedness that changes over time as a result of the first layer of each sub-network including the independent LSTMs.
3. The method of claim 1 , wherein the TLASM learns sub-network connections between different layers for information routing.
4. The method of claim 3 , wherein the TLASM models long-term dependency within a sequence of the historical trends.
5. The method of claim 4 , wherein a task-specific sub-network is implemented to integrate the long-term dependency and the short-term dependency.
6. The method of claim 1 , wherein a number of attentions used between a first layer and a second layer equals to a number of sub-networks in the second layer.
7. The method of claim 1 , wherein the attention mechanism is given as:
α
ij
=
exp
{
w
i
T
tan
h
(
V
i
h
t
(
j
)
)
}
∑
k
=
1
q
exp
{
w
i
T
tan
h
(
V
i
h
t
(
k
)
)
}
where W i ∈R d a and V i ∈R d a ×d are parameters and h represents hidden state vectors.
8. A non-transitory computer-readable storage medium comprising a computer-readable program for executing a multi-task deep learning model for learning trends in multivariate time series, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
collecting multi-variate time series data from a plurality of sensors;
jointly learning both local and global contextual features for predicting a trend of the multivariate time series by employing a tensorized long short-term memory (LSTM) with adaptive shared memory (TLASM) to learn historical dependency of historical trends, the adaptive shared memory including multiple layers of sub-networks, wherein the sub-networks are connected in parallel with each other, a first layer of each sub-network including multiple independent LSTMs followed by multiple layers of multilayer perceptrons (MLPs), and a last layer of the adaptive shared memory including task-specific MLPs, wherein all the independent LSTMs and the sub-networks are shared by all prediction tasks, and a connection between the sub-networks is a weighted average with an attention mechanism;
employing a multi-task one-dimensional convolutional neural network (1dCNN) to extract salient features from local raw time series data to model a short-term dependency between local time series data and subsequent trends, wherein each of the independent LSTMs directly communicates with each MLP of a first layer of MLPs of the multiple layers of MLPs; and
concatenating outputs of the TLASM and the multi-task one-dimensional convolutional neural network and the concatenated output are fed into a task-specific sub-network, wherein the task-specific sub-network generates trend prediction for the multivariate time series data and wherein the task-specific subnetwork includes an MLP layer.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the adaptive shared memory models task relatedness that changes over time as a result of the first layer of each sub-network including the independent LSTMs.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the TLASM learns sub-network connections between different layers for information routing.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the TLASM models long-term dependency within a sequence of the historical trends.
12. The non-transitory computer-readable storage medium of claim 11 , wherein a task-specific sub-network is implemented to integrate the long-term dependency and the short-term dependency.
13. The non-transitory computer-readable storage medium of claim 8 , wherein a number of attentions used between a first layer and a second layer equals to a number of sub-networks in the second layer.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the attention mechanism is given as:
α
ij
=
exp
{
w
i
T
tan
h
(
V
i
h
t
(
j
)
)
}
∑
k
=
1
q
exp
{
w
i
T
tan
h
(
V
i
h
t
(
k
)
)
}
where W i ∈R d a and V i ∈R d a ×d are parameters and h represents hidden state vectors.
15. A system for executing a multi-task deep learning model for learning trends in multivariate time series, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
collect multi-variate time series data from a plurality of sensors;
jointly learn both local and global contextual features for predicting a trend of the multivariate time series by employing a tensorized long short-term memory (LSTM) with adaptive shared memory (TLASM) to learn historical dependency of historical trends, the adaptive shared memory including multiple layers of sub-networks, wherein the sub-networks are connected in parallel with each other, a first layer of each sub-network including multiple independent LSTMs followed by multiple layers of multilayer perceptrons (MLPs), and a last layer of the adaptive shared memory including task-specific MLPs, wherein all the independent LSTMs and the sub-networks are shared by all prediction tasks, and a connection between the sub-networks is a weighted average with an attention mechanism;
employ a multi-task one-dimensional convolutional neural network (1dCNN) to extract salient features from local raw time series data to model a short-term dependency between local time series data and subsequent trends, wherein each of the independent LSTMs directly communicates with each MLP of a first layer of MLPs of the multiple layers of MLPs;
concatenate outputs of the TLASM and the multi-task one-dimensional convolutional neural network and the concatenated output are fed into a task-specific sub-network, wherein the task-specific sub-network generates trend prediction for the multivariate time series data and wherein the task-specific subnetwork includes an MLP layer.
16. The system of claim 15 , wherein the adaptive shared memory models task relatedness that changes over time as a result of the first layer of each sub-network including the independent LSTMs.
17. The system of claim 15 , wherein the TLASM learns sub-network connections between different layers for information routing.
18. The system of claim 17 , wherein the TLASM models long-term dependency within a sequence of the historical trends.
19. The system of claim 18 , wherein a task-specific sub-network is implemented to integrate the long-term dependency and the short-term dependency.
20. The system of claim 15 , wherein a number of attentions used between a first layer and a second layer equals to a number of sub-networks in the second layer.