Variational auto-encoder for multivariate time series generation
Certain aspects of the present disclosure provide techniques for generating multivariate time series data utilizing a variational auto-encoder (VAE) having an architecture for injecting custom temporal structures into the generated multivariate time series data. A method for generating multivariate time series data includes sampling a multivariate distribution forming a latent space vector, processing the latent space vector with an interpretable decoder of a variational auto-encoder, an architecture of the interpretable decoder comprising a plurality of blocks including one or more blocks configured to inject one or more temporal structures into multivariate time series data, and outputting, from the interpretable decoder, generated multivariate time series data comprising one or more temporal structures defined by the injected one or more temporal structures.
1 . A method for generating multivariate time series data, the method comprising:
sampling a multivariate distribution forming a latent space vector;
processing the sampled latent space vector with an interpretable decoder of a variational auto-encoder, an architecture of the interpretable decoder comprising a plurality of blocks including one or more temporal structure processing blocks and a base decoder block, wherein the plurality of blocks are configured to inject one or more temporal structures into multivariate time series data, and wherein the base decoder block is configured as one or more convolutional neural networks comprising a fully-connected linear layer, a reshaping layer, and a series of transposed convolutional layers; and
outputting, from the interpretable decoder, generated multivariate time series data comprising the one or more temporal structures defined by the injected one or more temporal structures.
2 . The method of claim 1 , wherein:
the plurality of blocks are configured to be selectively enabled or disabled, and
outputs from each enabled block of the plurality of blocks are added together to generate the multivariate time series data.
3 . The method of claim 1 , wherein:
at least one of the one or more temporal structure processing blocks is a trend block,
the trend block defines a monotonic function and the trend block comprises one or more fully-connected linear layers, a reshaping operation, and a trend reconstruction, and
when the trend block is enabled, the trend block imparts a monotonic temporal structure in the generated multivariate time series data.
4 . The method of claim 1 , wherein:
at least one of the one or more temporal structure processing blocks is a seasonality block,
the seasonality block defines a seasonal pattern parametrized at least by a number of seasons where each season has a predefined duration, and
the seasonal pattern is based on one or more elements of a matrix of basis expansion coefficients and one or more corresponding seasonal pattern values for the one or more elements.
5 . The method of claim 1 , wherein:
the one or more temporal structure processing blocks comprise a trend block and two or more seasonality blocks, and
a first seasonality block is distinct from a second seasonality block of the two or more seasonality blocks.
6 . The method of claim 1 , wherein the sampled latent space vector comprises a length equivalent to a dimensional size of the multivariate distribution.
7 . The method of claim 1 , wherein the plurality of blocks are configured as parallel blocks such that outputs from each of the plurality of blocks are added together to generate the multivariate time series data.
8 . The method of claim 1 , further comprising encoding, with an encoder of the variational auto-encoder, a signal comprising a 3-dimensional array of size N×T×D, where N is a batch size, T is a number of time steps, and D is a number of feature dimensions forming the multivariate distribution, wherein the encoder is configured as one or more convolutional neural networks.
9 . A computer-implemented system for generating multivariate time series data, the computer-implemented system comprising:
a processor and a non-transitory computer-readable medium storing computer readable instructions that, when executed by the processor, cause the system to:
sample a multivariate distribution forming a latent space vector;
process the sampled latent space vector with an interpretable decoder of a variational auto-encoder, an architecture of the interpretable decoder comprising a plurality of blocks including one or more temporal structure processing blocks and a base decoder block, wherein the plurality of blocks are configured to inject one or more temporal structures into a multivariate time series data, and wherein the base decoder block is configured as one or more convolutional neural networks comprising a fully-connected linear layer, a reshaping layer, and a series of transposed convolutional layers; and
output, from the interpretable decoder, generated multivariate time series data comprising one or more temporal structures defined by the injected one or more temporal structures.
10 . The computer-implemented system of claim 9 , wherein:
at least one of the one or more temporal structure processing blocks is a trend block,
the trend block defines a monotonic function and the trend block comprises one or more fully-connected linear layers, a reshaping operation, and a trend reconstruction, and
when the trend block is enabled, the trend block imparts a monotonic temporal structure in the generated multivariate time series data.
11 . The computer-implemented system of claim 9 , wherein:
at least one of the one or more temporal structure processing blocks is a seasonality block,
the seasonality block defines a seasonal pattern parametrized by a number of seasons where each season has a predefined duration, and
the seasonal pattern is based on one or more elements of a matrix of basis expansion coefficients and one or more corresponding seasonal pattern values for the one or more elements.
12 . The computer-implemented system of claim 9 , wherein:
at least one of the one or more temporal structure processing blocks is a concept drift block, and
the concept drift block defines a context shift in the multivariate time series data as a predefined shift in a mean or standard deviation of the multivariate time series data.
13 . The computer-implemented system of claim 9 , wherein:
the one or more temporal structure processing blocks comprise a trend block and two or more seasonality blocks, and
a first seasonality block is distinct from a second seasonality block of the two or more seasonality blocks.
14 . The computer-implemented system of claim 9 , wherein the sampled latent space vector comprises a length equivalent to a dimensional size of the multivariate distribution.
15 . The computer-implemented system of claim 9 , wherein the plurality of blocks are configured as parallel blocks such that outputs from each of the plurality of blocks are added together to generate the multivariate time series data.
16 . A method of training a variational auto-encoder for generating multivariate time series data, the method comprising:
inputting a plurality of data into an encoder of the variational auto-encoder, wherein the encoder is configured as one or more convolutional neural networks;
generating a multivariate distribution with the encoder;
sampling the multivariate distribution forming a latent space vector;
processing the sampled latent space vector with an interpretable decoder of the variational auto-encoder, wherein an architecture of the interpretable decoder comprises a plurality of blocks including one or more trainable temporal structure processing blocks and a base decoder block, wherein the plurality of blocks are configured to learn temporal structures in the sampled multivariate distribution, and wherein the base decoder block is configured as one or more convolutional neural networks comprising a fully-connected linear layer, a reshaping layer, and a series of transposed convolutional layers;
outputting, from the interpretable decoder, a generated multivariate time series data;
comparing the generated multivariate time series data with the plurality of data to determine a reconstruction error; and
backpropagating the determined reconstruction error to update the variational auto-encoder.
17 . The method of claim 16 , wherein the at least one of the one or more trainable temporal structure processing blocks is a trend block, the trend block learns a monotonic function and the trend block comprises one or more fully-connected linear layers, a reshaping operation, and a trend reconstruction.
18 . The method of claim 16 , wherein:
at least one of the one or more trainable temporal structure processing blocks is a seasonality block,
the seasonality block learns a seasonal pattern parametrized at least by a number of seasons where each season has a predefined duration, and
the seasonal pattern is based on one or more elements of a matrix of basis expansion coefficients and one or more corresponding seasonal pattern values for the one or more elements.
19 . The method of claim 18 , wherein an initial instance of the seasonality block is predefined based on at least one of an expected seasonal trend or a statistically determined cyclical pattern within the plurality of data.
20 . The method of claim 16 , wherein:
the plurality of blocks includes a plurality of seasonality blocks, and
each of the plurality of seasonality blocks is distinct such that during training hyperparameter tuning increases or decreases a strength of particular ones of the plurality of seasonality blocks that correspond to patterns within the plurality of data.