Method and system for predicting discrete sequences using deep context tree weighting
Systems and methods use deep, convolutional neural networks over exponentially long history windows to learn alphabets for context tree weighting (CTW) for prediction. Known issues of depth and breadth in conventional context tree weighting predictions are addressed by the systems and methods. To deal with depth, the history can be broken into time windows, permitting the ability to look exponentially far back while having less information the further one looks back. To deal with breadth, a deep neural network classifier can be used to learn to map arbitrary length histories to a small output symbol alphabet. The sequence of symbols produced by such a classifier over the history windows would then become the input sequence to CTW.
1. An artificial intelligence system, comprising: a computing device including at least one processor, one or more data storage devices, and a non-transitory data storage medium interfaced with the at least one processor, the non-transitory data storage medium containing instructions that, when executed cause the at least one processor to:
save observed sensory sequence information, as an original alphabet, in a plurality of history windows, the plurality of history windows being reverse chronological history windows, wherein a size of the plurality of history windows increase exponentially, wherein the plurality of history windows do not overlap;
apply a function to the observed sensory sequence information in each history window, wherein the function maps the observed sensory sequence information into a fixed set of discrete classes, wherein the fixed set of discrete classes is the same fixed set of discrete classes across each of the plurality of history windows;
perform a temporal convolution in a deep neural network to map observed sensory sequence information from the plurality of history windows to symbols,
wherein the temporal convolution includes defining each of the plurality of history windows of events as a ($2{circumflex over ( )}k$)-by-n matrix, where $2{circumflex over ( )}k$ is a number of time steps in each of the plurality of history windows and n is a number of events at each of the time steps, and applying a convolution that is an 1-by-n matrix, where 1 is less than 2{circumflex over ( )}k, wherein a set of the convolutions produces a new set of events;
apply a context tree weighting algorithm to an abstract alphabet resulting from the fixed set of discrete classes for each of the plurality of history windows, wherein the abstract alphabet is smaller in size than the original alphabet; and
make a prediction in the original alphabet based on patterns in the abstract alphabet.
2. The artificial intelligence system of claim 1 , wherein the instructions cause the at least one processor to choose at least one hyperparameter for each of the plurality of history window to allow the system to trade off bias-variance.
3. The artificial intelligence system of claim 1 , wherein the function is a feature-wise maximum over time steps in one or more of the plurality of history windows.
4. The artificial intelligence system of claim 3 , wherein the observed sensory sequence information is a binary event.
5. The artificial intelligence system of claim 1 , wherein the instructions cause the at least one processor to use a deep neural network classifier to map arbitrary length histories to a second alphabet having smaller length than the alphabet of the arbitrary length histories as an input sequence for the context tree weighting algorithm.
6. The artificial intelligence system of claim 5 , wherein a long short-term memory-based sequence to symbol method is used to map the arbitrary length histories to the second alphabet having smaller length than the alphabet of the arbitrary length histories.
7. The artificial intelligence system of claim 1 , wherein the convolution is applied to each of the plurality of history windows.
8. An artificial intelligence system, comprising: a computing device including at least one processor, one or more data storage devices, and a non-transitory data storage medium interfaced with the at least one processor, the non-transitory data storage medium containing instructions that, when executed cause the at least one processor to:
save observed sensory sequence information, as an original alphabet, in a plurality of history windows, the plurality of history windows being reverse chronological history windows, wherein a size of the plurality of history windows increase exponentially, wherein the plurality of history windows do not overlap;
apply a function to the observed sensory sequence information in each history window, wherein the function maps the observed sensory sequence information into a fixed set of discrete classes, wherein the fixed set of discrete classes is the same fixed set of discrete classes across each of the plurality of history windows;
choose at least one hyperparameter for each of the plurality of history window to allow the system to trade off bias-variance;
use a deep neural network classifier to map arbitrary length histories to an abstract alphabet having smaller length than the original alphabet of the arbitrary length history windows as an input sequence for the context tree weighting algorithm;
apply a context tree weighting algorithm to the abstract alphabet resulting from the fixed set of discrete classes for each of the plurality of history windows;
perform a temporal convolution in a deep neural network to map observed sensory sequence information from the plurality of history windows to symbols,
wherein the temporal convolution includes defining each of the plurality of history windows of events as a ($2{circumflex over ( )}k$)-by-n matrix, where $2{circumflex over ( )}k$ is a number of time steps in each of the plurality of history windows and n is a number of events at each of the time steps, and applying a convolution that is an 1-by-n matrix, where 1 is less than 2{circumflex over ( )}k, wherein a set of the convolutions produces a new set of events; and
make a prediction in the original alphabet based on patterns in the abstract alphabet.
9. The artificial intelligence system of claim 8 , wherein a long short-term memory-based sequence to symbol method is used to map the arbitrary length histories to the second alphabet having smaller length than the alphabet of the arbitrary length histories.
10. The artificial intelligence system of claim 8 , wherein the observed sensory sequence information is a binary event.
11. An artificial intelligence system, comprising: a computing device including at least one processor, one or more data storage devices, and a non-transitory data storage medium interfaced with the at least one processor, the non-transitory data storage medium containing instructions that, when executed cause the at least one processor to:
save observed sensory sequence information, as an original alphabet, in a plurality of history windows, the plurality of history windows being reverse chronological history windows, wherein a size of the plurality of history windows increase exponentially, wherein the plurality of history windows do not overlap;
apply a function to the observed sensory sequence information in each history window, wherein the function maps the observed sensory sequence information into a fixed set of discrete classes, wherein the fixed set of discrete classes is the same fixed set of discrete classes across each of the plurality of history windows;
choose at least one hyperparameter for each of the plurality of history window to allow the system to trade off bias-variance;
perform a temporal convolution in a deep neural network to map observed sensory sequence information from the plurality of history windows to the abstract alphabet, wherein the temporal convolution includes defining each of the plurality of history windows of events as a ($2{circumflex over ( )}k$)-by-n matrix, where $2{circumflex over ( )}k$ is a number of time steps in each of the plurality of history windows and n is a number of events at each of the time steps, and applying a convolution that is an 1-by-n matric, where 1 is less than 2{circumflex over ( )}k, wherein a set of the convolutions produces a new set of events, wherein the set of the convolutions includes a predetermined number of convolution layers, wherein the temporal convolution determines a maximum of the resulting events and ensures the fixed set of discrete classes is independent from the size of each of the plurality of history windows;
apply a context tree weighting algorithm to the abstract alphabet resulting from the fixed set of discrete classes for each of the plurality of history windows to predict a future discrete sequence, wherein the abstract alphabet is smaller in size than the original alphabet; and
make a prediction in the original alphabet based on patterns in the abstract alphabet.