Decoder-only transformer model for time series data
A system and method include forecasting a series of future data points in a long sequence time series data using a decoder-only transformer model by dividing the long sequence time series data into a plurality of sequences, converting each sequence of the plurality of sequences into a first vector to obtain a plurality of first vectors, creating a plurality of second vectors from the time stamps associated with the plurality of data points, combining the first vector with the second vector of each sequence of the plurality of sequences to obtain a plurality of third vectors, computing a context matrix from the plurality of third vectors, performing a convolution operation on the context matrix to forecast the series of future data points, and outputting the series of future data points from the prediction layer.
1. A non-transitory computer-readable medium comprising computer-readable instructions that when executed by a processor cause the processor to:
receive a long sequence time series data, the long sequence time series data comprising a plurality of data points, each data point of the plurality of data points associated with a time stamp; and
forecast a series of future data points in the long sequence time series data using a decoder-only transformer model by:
creating an embedding for the long sequence time series data in an embedding layer of the decoder-only transformer model by:
dividing the long sequence time series data into a plurality of sequences, each sequence of the plurality of sequences having consecutive n data points of the plurality of data points, wherein each sequence of the plurality of sequences is offset from a neighboring sequence of the plurality of sequences based on a shift window;
converting each sequence of the plurality of sequences into a first vector to obtain a plurality of first vectors;
creating a plurality of second vectors from the time stamps associated with the plurality of data points, wherein each second vector of the plurality of second vectors corresponds to one sequence of the plurality of sequences; and
combining the first vector with the second vector of each sequence of the plurality of sequences to obtain a plurality of third vectors, wherein the plurality of third vectors corresponds to the embedding;
computing a context matrix in a decoder layer of the decoder-only transformer model based on the embedding;
inputting the context matrix into a prediction layer of the decoder-only transformer model;
performing a convolution operation on the context matrix to forecast the series of future data points; and
outputting the series of future data points from the prediction layer.
2. The non-transitory computer-readable medium of claim 1 , wherein to divide the long sequence time series data into the plurality of sequences, the computer-readable instructions further cause the processor to:
(A) define the shift window to indicate an overlap between consecutive sequences of the plurality of sequences;
(B) generate a current sequence of the plurality of sequences from the long sequence time series data;
(C) determine a starting data point for a next sequence of the plurality of sequences based on the shift window and a starting data point of the current sequence, wherein the current sequence and the next sequence are consecutive, and wherein the starting point of the next sequence is offset from the starting point of the current sequence by the shift window; and
(D) repeat (C) with the next sequence as the current sequence until the long sequence time series data is divided into the plurality of sequences.
3. The non-transitory computer-readable medium of claim 2 , wherein the shift window is between and including one data point and n data points.
4. The non-transitory computer-readable medium of claim 2 , wherein the overlap between two neighboring sequences of the plurality of sequences is between and including zero and n−1 data points.
5. The non-transitory computer-readable medium of claim 1 , wherein to convert each sequence of the plurality of sequences into the first vector, the computer-readable instructions further cause the processor to perform a linear transformation on the sequence using an nn.linear linear transformation function.
6. The non-transitory computer-readable medium of claim 1 , wherein to create each second vector of the plurality of second vectors, the computer-readable instructions further cause the processor to:
create a Day of Week vector from the timestamps associated with data points in each sequence of the plurality of sequences;
create a Week of Year vector from the timestamps associated with data points in each sequence of the plurality of sequences;
create an Hour of Day vector from the timestamps associated with data points in each sequence of the plurality of sequences;
perform an nn.linear linear transformation on each of the Day of Week vector, the Week of Year vector, and the Hour of Day vector to generate a Day of Week linearly transformed vector, a Week of Year linearly transformed vector, and a Day of Week linearly transformed vector, respectively; and
aggregate the Day of Week linearly transformed vector, the Week of Year linearly transformed vector, and the Day of Week linearly transformed vector to generate the second vector.
7. The non-transitory computer-readable medium of claim 1 , wherein to combine the first vector and the second vector of each of the plurality of sequences to obtain the plurality of third vectors, the computer-readable instructions further cause the processor to perform a matrix addition of the first vector and the second vector of each of the plurality of sequences.
8. The non-transitory computer-readable medium of claim 1 , wherein the first vector, the second vector, and the third vector of each of the plurality of sequences is of a same vector size.
9. The non-transitory computer-readable medium of claim 1 , wherein the decoder layer comprises a plurality of stacked decoders, and wherein each of the plurality of stacked decoders comprises:
the multi-head attention mechanism that receives the plurality of third vectors or an output from a previous decoder;
a feed forward layer;
a first normalization layer between the multi-head attention mechanism and the feed forward layer; and
a second normalization layer that receives an output of the feed forward layer, wherein the output from a last decoder of the plurality of stacked decoders is the context matrix.
10. The non-transitory computer-readable medium of claim 1 , wherein the convolution operation is a 1D-convolution operation.
11. The non-transitory computer-readable medium of claim 1 , wherein the decoder-only transformer model forecasts all future data points in the series of future data points in parallel.
12. A system comprising:
a decoder-only transformer model comprising:
an embedding layer;
a decoder layer comprising a plurality of stacker decoders; and
a prediction layer;
a memory having computer-readable instructions stored thereon; and
a processor that executes the computer-readable instructions to:
receive a long sequence time series data, the long sequence time series data comprising a plurality of data points, each data point of the plurality of data points associated with a time stamp; and
forecast a series of future data points in the long sequence time series data using the decoder-only transformer model by:
creating an embedding for the long sequence time series data in the embedding layer of the decoder-only transformer model by:
dividing the long sequence time series data into a plurality of sequences, each sequence of the plurality of sequences having consecutive n data points of the plurality of data points, wherein each sequence of the plurality of sequences is offset from a neighboring sequence of the plurality of sequences based on a shift window;
converting each sequence of the plurality of sequences into a first vector to obtain a plurality of first vectors;
creating a plurality of second vectors from the time stamps associated with the plurality of data points, wherein each second vector of the plurality of second vectors corresponds to one sequence of the plurality of sequences; and
combining the first vector with the second vector of each sequence of the plurality of sequences to obtain a plurality of third vectors, wherein the plurality of third vectors corresponds to the embedding;
computing a context matrix in the decoder layer based on the embedding;
inputting the context matrix into the prediction layer;
performing a convolution operation on the context matrix to forecast the series of future data points; and
outputting the series of future data points from the prediction layer.
13. The system of claim 12 , wherein to divide the long sequence time series data into the plurality of sequences, the computer-readable instructions further cause the processor to:
(A) define the shift window to indicate an overlap between consecutive sequences of the plurality of sequences;
(B) generate a current sequence of the plurality of sequences from the long sequence time series data;
(C) determine a starting data point for a next sequence of the plurality of sequences based on the shift window and a starting data point of the current sequence, wherein the current sequence and the next sequence are consecutive, and wherein the starting point of the next sequence is offset from the starting point of the current sequence by the shift window; and
(D) repeat (C) with the next sequence as the current sequence until the long sequence time series data is divided into the plurality of sequences.
14. The system of claim 13 , wherein the shift window is between and including one data point and n data points.
15. The system of claim 13 , wherein the overlap between two neighboring sequences of the plurality of sequences is between and including zero and n−1 data points.
16. The system of claim 12 , wherein to convert each sequence of the plurality of sequences into the first vector, the computer-readable instructions further cause the processor to perform a linear transformation on the sequence using an nn.linear linear transformation function.
17. The system of claim 12 , wherein to create each second vector of the plurality of second vectors, the computer-readable instructions further cause the processor to:
create a Day of Week vector from the timestamps associated with data points in each sequence of the plurality of sequences;
create a Week of Year vector from the timestamps associated with data points in each sequence of the plurality of sequences;
create an Hour of Day vector from the timestamps associated with data points in each sequence of the plurality of sequences;
perform an nn.linear linear transformation on each of the Day of Week vector, the Week of Year vector, and the Hour of Day vector to generate a Day of Week linearly transformed vector, a Week of Year linearly transformed vector, and a Day of Week linearly transformed vector, respectively; and
aggregate the Day of Week linearly transformed vector, the Week of Year linearly transformed vector, and the Day of Week linearly transformed vector to generate the second vector.
18. The system of claim 12 , wherein to combine the first vector and the second vector of each of the plurality of sequences to obtain the plurality of third vectors, the computer-readable instructions further cause the processor to perform a matrix addition of the first vector and the second vector of each of the plurality of sequences.
19. The system of claim 12 , wherein the first vector, the second vector, and the third vector of each of the plurality of sequences is of a same vector size.
20. The system of claim 12 , wherein the decoder layer comprises a plurality of stacked decoders, and wherein each of the plurality of stacked decoders comprises:
the multi-head attention mechanism that receives the plurality of third vectors or an output from a previous decoder;
a feed forward layer;
a first normalization layer between the multi-head attention mechanism and the feed forward layer; and
a second normalization layer that receives an output of the feed forward layer, wherein the output from a last decoder of the plurality of stacked decoders is the context matrix.
21. The system of claim 12 , wherein the convolution operation is a 1D-convolution operation.
22. The system of claim 12 , wherein the decoder-only transformer model forecasts all future data points in the series of future data points in parallel.
23. A method comprising:
receiving, by a processor executing computer-readable instructions stored on a memory, a long sequence time series data, the long sequence time series data comprising a plurality of data points, each data point of the plurality of data points associated with a time stamp; and
forecasting, by the processor, a series of future data points in the long sequence time series data using a decoder-only transformer model by:
creating an embedding for the long sequence time series data in an embedding layer of the decoder-only transformer model by:
dividing, by the processor, the long sequence time series data into a plurality of sequences, each sequence of the plurality of sequences having consecutive n data points of the plurality of data points, wherein each sequence of the plurality of sequences is offset from a neighboring sequence of the plurality of sequences based on a shift window;
converting, by the processor, each sequence of the plurality of sequences into a first vector to obtain a plurality of first vectors;
creating, by the processor, a plurality of second vectors from the time stamps associated with the plurality of data points, wherein each second vector of the plurality of second vectors corresponds to one sequence of the plurality of sequences; and
combining, by the processor, the first vector with the second vector of each sequence of the plurality of sequences to obtain a plurality of third vectors, wherein the plurality of third vectors corresponds to the embedding;
computing, by the processor, a context matrix in a decoder layer of the decoder-only transformer model based on the embedding;
inputting, by the processor, the context matrix into a prediction layer of the decoder-only transformer model;
performing, by the processor, a convolution operation on the context matrix to forecast the series of future data points; and
outputting, by the processor, the series of future data points from the prediction layer.
24. The method of claim 23 , wherein to divide the long sequence time series data into the plurality of sequences, the method further comprises:
(A) defining, by the processor, the shift window to indicate an overlap between consecutive sequences of the plurality of sequences;
(B) generating, by the processor, a current sequence of the plurality of sequences from the long sequence time series data;
(C) determining, by the processor, a starting data point for a next sequence of the plurality of sequences based on the shift window and a starting data point of the current sequence, wherein the current sequence and the next sequence are consecutive, and wherein the starting point of the next sequence is offset from the starting point of the current sequence by the shift window; and
(D) repeating, by the processor, (C) with the next sequence as the current sequence until the long sequence time series data is divided into the plurality of sequences.
25. The method of claim 24 , wherein the shift window is between and including one data point and n data points.
26. The method of claim 24 , wherein the overlap between two neighboring sequences of the plurality of sequences is between and including zero and n−1 data points.
27. The method of claim 23 , wherein to convert each sequence of the plurality of sequences into the first vector, the method further comprises performing, by the processor, a linear transformation on the sequence using an nn.linear linear transformation function.
28. The method of claim 23 , wherein to create each second vector of the plurality of second vectors, the method further comprises:
creating, by the processor, a Day of Week vector from the timestamps associated with data points in each sequence of the plurality of sequences;
creating, by the processor, a Week of Year vector from the timestamps associated with data points in each sequence of the plurality of sequences;
creating, by the processor, an Hour of Day vector from the timestamps associated with data points in each sequence of the plurality of sequences;
performing, by the processor, an nn.linear linear transformation on each of the Day of Week vector, the Week of Year vector, and the Hour of Day vector to generate a Day of Week linearly transformed vector, a Week of Year linearly transformed vector, and a Day of Week linearly transformed vector, respectively; and
aggregating, by the processor, the Day of Week linearly transformed vector, the Week of Year linearly transformed vector, and the Day of Week linearly transformed vector to generate the second vector.
29. The method of claim 23 , wherein to combine the first vector and the second vector of each of the plurality of sequences to obtain the plurality of third vectors, the method further comprises performing, by the processor, a matrix addition of the first vector and the second vector of each of the plurality of sequences.
30. The method of claim 23 , wherein the decoder layer comprises a plurality of stacked decoders, and wherein each of the plurality of stacked decoders comprises:
the multi-head attention mechanism that receives the plurality of third vectors or an output from a previous decoder;
a feed forward layer;
a first normalization layer between the multi-head attention mechanism and the feed forward layer; and
a second normalization layer that receives an output of the feed forward layer, wherein the output from a last decoder of the plurality of stacked decoders is the context matrix.