Temporal sequence causal transformer machine learning model
Various embodiments of the present disclosure provide methods, apparatus, systems, computing devices, computing entities, and/or the like for generating a prediction output comprising one or more actions by receiving data associated with encounters in a tuple form, tokenizing the encounters, training a causal transformer machine learning model configured to predict outcomes of actions by translating action tokens from the tokenized encounters into one or more embedding spaces, and training a causal transformer machine learning model to select the one or more actions based on embeddings from the one or more embedding spaces.
1 . A computer-implemented method comprising:
receiving, by one or more processors, an input temporal sequence, wherein:
(i) the input temporal sequence comprises a set of one or more input tuples,
(ii) an input tuple of the set of one or more input tuples comprises a plurality of tuple data objects comprising data representative of (a) one or more states, (b) one or more combinations of actions, (c) one or more outcomes, and (d) a cumulative discounted future outcome, and
(iii) the cumulative discounted future outcome is associated with the input tuple;
generating, by the one or more processors, a plurality of input tokens associated with the plurality of tuple data objects, wherein the plurality of input tokens are generated according to a plurality of respective tuple data object types associated with the plurality of tuple data objects;
generating, by the one or more processors and using a causal transformer machine learning model, a prediction output based on the plurality of input tokens and a conditional distribution of actions, the prediction output comprising a plurality of output tokens, wherein training the causal transformer machine learning model comprises:
(a) projecting a plurality of training tokens into a plurality of respective embedding spaces using an embedding layer, wherein (i) at least one of the plurality of respective embedding spaces comprises a plurality of embedding sets associated with the plurality of training tokens, (ii) the plurality of embedding sets comprises a temporal embedding set, a structural embedding set, and a positional embedding set, (iii) the plurality of training tokens is associated with a plurality of training temporal sequences, and (iv) a training temporal sequence of the plurality of training temporal sequences comprises a set of one or more training tuples, wherein a training tuple of the set of one or more training tuples comprises a plurality of training tuple data objects comprising data representative of one or more training states, one or more training combinations of actions, one or more training outcomes, and a training cumulative discounted future outcome associated with the training tuple,
(b) inputting the plurality of respective embedding spaces into the causal transformer machine learning model, and
(c) for the training tuple, generating a context dependent representation based on one or more of the plurality of respective embedding spaces associated with one or more sequentially prior tuples of the set of one or more training tuples with respect to the training tuple;
generating, by the one or more processors, one or more policy scores based on the prediction output; and
initiating, by the one or more processors, the performance of one or more prediction-based actions based on the one or more policy scores and the prediction output.
2 . The computer-implemented method of claim 1 , wherein the temporal embedding set comprises one or more embeddings associated with a relative time between the training tuple and a sequentially first training tuple of the set of one or more training tuples.
3 . The computer-implemented method of claim 1 , wherein the structural embedding set comprises one or more embeddings associated with the plurality of respective tuple data object types of the plurality of training tuple data objects.
4 . The computer-implemented method of claim 1 , wherein the positional embedding set comprises one or more embeddings associated with a sequential position of the training tuple.
5 . The computer-implemented method of claim 1 further comprising discarding at least one sequentially first training tuple of the set of one or more training tuples from a subset of training temporal sequences of the plurality of training temporal sequences.
6 . The computer-implemented method of claim 1 , wherein the causal transformer machine learning model is trained based on teacher-forcing training by using one or more ground-truth tokens as training feedback input to the causal transformer machine learning model.
7 . The computer-implemented method of claim 1 , wherein the plurality of output tokens comprises one or more output state tokens, one or more output action tokens, one or more output outcome tokens, and one or more output cumulative discounted future outcome tokens.
8 . The computer-implemented method of claim 7 , wherein generating the prediction output further comprises generating one or more log- likelihood scores of one or more output actions associated with the one or more output action tokens, the one or more log-likelihood scores representative of a likelihood of the one or more output actions most likely to follow based on the input temporal sequence.
9 . The computer-implemented method of claim 7 , wherein generating the prediction output further comprises generating one or more predictive scores, the one or more predictive scores comprising (i) one or more action predictive scores of one or more output actions associated with the one or more output action tokens based on the one or more states, and (ii) one or more outcome predictive scores associated with one or more output cumulative discounted future outcomes associated with the one or more output cumulative discounted future outcome tokens based on the one or more output actions.
10 . The computer-implemented method of claim 9 , wherein generating the prediction output further comprises generating one or more expected predicted outcomes based on the one or more output cumulative discounted future outcomes and the one or more predictive scores.
11 . The computer-implemented method of claim 7 , wherein initiating the performance of the one or more prediction-based actions further comprises selecting one or more output actions associated with the one or more output action tokens based on the one or more policy scores.
12 . The computer-implemented method of claim 1 , further comprising excluding one or more action combination tokens of a plurality of action combination tokens from the conditional distribution of actions based on the one or more action combination tokens comprising one or more probability scores that are below a threshold.
13 . The computer-implemented method of claim 1 , wherein generating the plurality of input tokens further comprises:
receiving an action space data object comprising a plurality of possible individual actions; and
assigning a plurality of action combination tokens to a plurality of combinations comprising selected ones a subset of possible individual actions of the plurality of possible individual actions.
14 . The computer-implemented method of claim 1 further comprising generating the conditional distribution of actions based on the one or more states.
15 . A system comprising one or more processors and
one or more non-transitory computer readable media storing processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving an input temporal sequence, wherein:
(i) the input temporal sequence comprises a set of one or more input tuples,
(ii) an input tuple of the set of one or more input tuples comprises a plurality of tuple data objects comprising data representative of (a) one or more states, (b) one or more combinations of actions, (c) one or more outcomes, and (d) a cumulative discounted future outcome, and
(iii) the cumulative discounted future outcome is associated with the input tuple;
generating a plurality of input tokens associated with the plurality of tuple data objects, wherein the plurality of input tokens are generated according to a plurality of respective tuple data object types associated with the plurality of tuple data objects;
generating, using a causal transformer machine learning model, a prediction output based on the plurality of input tokens and a conditional distribution of actions, the prediction output comprising a plurality of output tokens, wherein training the causal transformer machine learning model comprises:
(a) projecting a plurality of training tokens into a plurality of respective embedding spaces using an embedding layer, wherein (i) at least one of the plurality of respective embedding spaces comprises a plurality of embedding sets associated with the plurality of training tokens, (ii) the plurality of embedding sets comprises a temporal embedding set, a structural embedding set, and a positional embedding set, (iii) the plurality of training tokens is associated with a plurality of training temporal sequences, and (iv) a training temporal sequence of the plurality of training temporal sequences comprises a set of one or more training tuples, wherein a training tuple of the set of one or more training tuples comprises a plurality of training tuple data objects comprising data representative of one or more training states, one or more training combinations of actions, one or more training outcomes, and a training cumulative discounted future outcome associated with the training tuple,
(b) inputting the plurality of respective embedding spaces into the causal transformer machine learning model, and
(c) for the training tuple, generating a context dependent representation based on one or more of the plurality of respective embedding spaces associated with one or more sequentially prior tuples of the set of one or more training tuples with respect to the training tuple;
generating one or more policy scores based on the prediction output; and
initiating the performance of one or more prediction-based actions based on the one or more policy scores and the prediction output.
16 . The system of claim 15 , wherein the plurality of output tokens comprises one or more output state tokens, one or more output action tokens, one or more output outcome tokens, and one or more output cumulative discounted future outcome tokens.
17 . The system of claim 16 , wherein the operations further comprise generating the prediction output by generating one or more log-likelihood scores of one or more output actions associated with the one or more output action tokens, the one or more log-likelihood scores representative of a likelihood of the one or more output actions most likely to follow based on the input temporal sequence.
18 . The system of claim 16 , wherein the operations further comprise generating the prediction output by generating one or more predictive scores, the one or more predictive scores comprising (i) one or more action predictive scores of one or more output actions associated with the one or more output action tokens based on the one or more states, and (ii) one or more outcome predictive scores associated with one or more output cumulative discounted future outcomes associated with the one or more output cumulative discounted future outcome tokens based on the one or more output actions.
19 . The system of claim 18 , wherein the operations further comprise generating the prediction output by generating one or more expected predicted outcomes based on the one or more output cumulative discounted future outcomes and the one or more predictive scores.
20 . One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving an input temporal sequence, wherein:
(i) the input temporal sequence comprises a set of one or more input tuples,
(ii) an input tuple of the set of one or more input tuples comprises a plurality of tuple data objects comprising data representative of (a) one or more states, (b) one or more combinations of actions, (c) one or more outcomes, and (d) a cumulative discounted future outcome, and
(iii) the cumulative discounted future outcome is associated with the input tuple;
generating a plurality of input tokens associated with the plurality of tuple data objects, wherein the plurality of input tokens are generated according to a plurality of respective tuple data object types associated with the plurality of tuple data objects;
generating, using a causal transformer machine learning model, a prediction output based on the plurality of input tokens and a conditional distribution of actions, the prediction output comprising a plurality of output tokens, wherein training the causal transformer machine learning model comprises:
(a) projecting a plurality of training tokens into a plurality of respective embedding spaces using an embedding layer, wherein (i) at least one of the plurality of respective embedding spaces comprises a plurality of embedding sets associated with the plurality of training tokens, (ii) the plurality of embedding sets comprises a temporal embedding set, a structural embedding set, and a positional embedding set, (iii) the plurality of training tokens is associated with a plurality of training temporal sequences, and (iv) a training temporal sequence of the plurality of training temporal sequences comprises a set of one or more training tuples, wherein a training tuple of the set of one or more training tuples comprises a plurality of training tuple data objects comprising data representative of one or more training states, one or more training combinations of actions, one or more training outcomes, and a training cumulative discounted future outcome associated with training tuple,
(b) inputting the plurality of respective embedding spaces into the causal transformer machine learning model, and
(c) for the training tuple, generating a context dependent representation based on one or more of the plurality of respective embedding spaces associated with one or more sequentially prior tuples of the set of one or more training tuples with respect to the training tuple;
generating one or more policy scores based on the prediction output; and
initiating the performance of one or more prediction-based actions based on the one or more policy scores and the prediction output.