Systems and methods for mutual information based self-supervised learning
Embodiments described herein combine both masked reconstruction and predictive coding. Specifically, unlike contrastive learning, the mutual information between past states and future states are directly estimated. The context information can also be directly captured via shifted masked reconstruction—unlike standard masked reconstruction, the target reconstructed observations are shifted slightly towards the future to incorporate more predictability. The estimated mutual information and shifted masked reconstruction loss can then be combined as the loss function to update the neural model.
1. A system for training a neural network model to generate a time series prediction, the system comprising:
a memory configured to store a neural network model comprising a bi-directional long short-term memory (LSTM) encoder and a decoder, and a plurality of processor-executable instructions;
an input interface configured to receive a time sequence of input data instances over a time window corresponding to a window size;
a processor configured to execute the plurality of processor-executable instructions to:
encode, by the bi-directional LSTM encoder of the neural network model implemented on one or more hardware processors, the time sequence of input data instances to a sequence of latent states in a latent space;
dividing the sequence of latent states into a past vector in the latent space and a future vector in the latent space separated by a first timestamp;
compute mutual information corresponding to the first timestamp between the past vector of a first subset of the sequence of latent states before the first timestamp and the future vector of a second subset of the sequence of latent states after the first timestamp in the latent space;
compute a mutual information objective for the time window by averaging the computed mutual information with the first timestamp taking a value across all timestamps within the time window;
train the neural network model of the bi-directional LSTM encoder and the decoder by updating parameters of the neural network model based on at least the mutual information objective via backpropagation; and
generate, by the decoder of the trained neural network model, an output time series prediction having a first sequence of tokens representing data points over a future time window in response to an input having a second sequence of tokens representing observations in a past time window.
2. The system of claim 1 , wherein the time sequence of input data instances are embedded by a feature extractor to a low-dimensional space, and
wherein each of the sequence of latent states has a size based on a dimension of the low-dimensional space and the window size.
3. The system of claim 1 , wherein the set of latent states form a Gaussian process.
4. The system of claim 3 , wherein the mutual information objective is computed by estimating an entropy of the past vector or the future vector in the latent space according to the Gaussian process.
5. The system of claim 1 , wherein the processor is further configured to:
randomly replace a subset of input data instances from the time sequence of input data instances with one or more mask tokens;
encode, via the encoder, a masked time sequence of input data instances into an encoded sequence of data representations in the latent space;
compute a masked reconstruction loss based on a comparison of a decoded sequence of data representations from the latent space and a future input sequence, wherein the future input sequence is obtained by right shifting the time sequence of input data instances for a number of timestamps.
6. The system of claim 5 , wherein the masked reconstruction loss is computed by further right shifting a binary mask for the number of timestamps and applying the right shifted binary mask to the comparison of the decoded sequence and the future input sequence.
7. The system of claim 5 , wherein the processor is further configured to:
compute a combined objective based on a weighted sum of the masked reconstruction loss and the mutual information objective; and
update the neural network model using the combined objective.
8. A method for training a neural network model to generate a time series prediction, the method comprising:
receiving, via a data interface, a time sequence of input data instances over a time window corresponding to a window size;
encoding, via a bi-directional long short-term memory (LSTM) encoder of the neural network model implemented on one or more hardware processors, the time sequence of input data instances to a sequence of latent states in a latent space;
generating a past vector in the latent space and a future vector in the latent space separated by a first timestamp based on the sequence of latent states;
computing mutual information corresponding to the first timestamp between the past vector of a first subset of the sequence of latent states before the first timestamp and the future vector of a second subset of the sequence of latent states after the first timestamp in the latent space;
computing a mutual information objective for the time window by averaging the computed mutual information with the first timestamp taking a value across all timestamps within the time window;
training the neural network model of the bi-directional LSTM encoder and the decoder by updating parameters of the neural network model based on at least the mutual information objective via backpropagation; and
generating, by the decoder of the trained neural network model, an output time series prediction having a first sequence of tokens representing data points over a future time window in response to an input having a second sequence of tokens representing observations in a past time window.
9. The method of claim 8 , wherein the time sequence of input data instances are embedded by a feature extractor to a low-dimensional space, and
wherein each of the sequence of latent states has a size based on a dimension of the low-dimensional space and the window size.
10. The method of claim 8 , wherein the set of latent states form a Gaussian process.
11. The method of claim 10 , wherein the mutual information objective is computed by estimating an entropy of the past vector or the future vector in the latent space according to the Gaussian process.
12. The method of claim 8 , further comprising:
randomly replacing a subset of input data instances from the time sequence of input data instances with one or more mask tokens;
encoding, via the encoder, a masked time sequence of input data instances into an encoded sequence of data representations in the latent space;
computing a masked reconstruction loss based on a comparison of a decoded sequence of data representations from the latent space and a future input sequence, wherein the future input sequence is obtained by right shifting the time sequence of input data instances for a number of timestamps.
13. The method of claim 12 , wherein the masked reconstruction loss is computed by further right shifting a binary mask for the number of timestamps and applying the right shifted binary mask to the comparison of the decoded sequence and the future input sequence.
14. The method of claim 12 , further comprising:
computing a combined objective based on a weighted sum of the masked reconstruction loss and the mutual information objective; and
updating the neural network model using the combined objective.
15. A non-transitory processor-readable storage medium storing processor-executable instructions for training a neural network model to generate a time series prediction, the instructions being executed by a processor to perform operations comprising:
receiving, via a data interface, a time sequence of input data instances over a time window corresponding to a window size;
encoding, via a bi-directional long short-term memory (LSTM) encoder of the neural network model implemented on one or more hardware processors, the time sequence of input data instances to a sequence of latent states in a latent space;
generating a past vector in the latent space and a future vector in the latent space separated by a first timestamp based on the sequence of latent states;
computing mutual information corresponding to the first timestamp between the past vector of a first subset of the sequence of latent states before the first timestamp and the future vector of a second subset of the sequence of latent states after the first timestamp in the latent space;
computing a mutual information objective for the time window by averaging the computed mutual information with the first timestamp taking a value across all timestamps within the time window;
training the neural network model of the bi-directional LSTM encoder and a decoder by updating parameters of the neural network model based on at least the mutual information objective via backpropagation; and
generating, by the decoder of the trained neural network model, an output time series prediction having a first sequence of tokens representing data points over a future time window in response to an input question having a second sequence of tokens representing observations in a past time window.
16. The medium of claim 15 , wherein the time sequence of input data instances are embedded by a feature extractor to a low-dimensional space, and
wherein each of the sequence of latent states has a size based on a dimension of the low-dimensional space and the window size.
17. The medium of claim 15 , wherein the set of latent states form a Gaussian process, and wherein the mutual information objective is computed by estimating an entropy of the past vector or the future vector in the latent space according to the Gaussian process.
18. The medium of claim 15 , wherein the operations further comprise:
randomly replacing a subset of input data instances from the time sequence of input data instances with one or more mask tokens;
encoding, via the encoder, a masked time sequence of input data instances into an encoded sequence of data representations in the latent space; and
computing a masked reconstruction loss based on a comparison of a decoded sequence of data representations from the latent space and a future input sequence, wherein the future input sequence is obtained by right shifting the time sequence of input data instances for a number of timestamps.
19. The medium of claim 18 , wherein the masked reconstruction loss is computed by further right shifting a binary mask for the number of timestamps and applying the right shifted binary mask to the comparison of the decoded sequence and the future input sequence.
20. The medium of claim 18 , wherein the operations further comprise:
computing a combined objective based on a weighted sum of the masked reconstruction loss and the mutual information objective; and
updating the neural network model using the combined objective.