Multi-turn dialogue response generation with autoregressive transformer models
Machine classifiers in accordance with embodiments of the invention capture long-term temporal dependencies in the dialogue data better than the existing RNN-based architectures. Additionally, machine classifiers may model the joint distribution of the context and response as opposed to the conditional distribution of the response given the context as employed in sequence-to-sequence frameworks. Machine classifiers in accordance with embodiments further append random paddings before and/or after the input data to reduce the syntactic redundancy in the input data, thereby improving the performance of the machine classifiers for a variety of dialogue-related tasks. The random padding of the input data may further provide regularization during the training of the machine classifier and/or reduce exposure bias. In a variety of embodiments, the input data may be encoded based on subword tokenization.
1. A computer-implemented method, comprising:
initializing a model having a sequence to sequence network architecture comprising an encoder and a decoder;
training the model using a plurality of training sequences, wherein each training sequence comprises an encoder sequence and a decoder sequence, and wherein training the model comprises:
generating an encoding for each training sequence in the plurality of training sequences; and
for each encoding:
randomly inserting an informative padding, comprising a random sampling of encoded tokens from the plurality of training sequences, into the encoder sequence of the encoding; and
training the model using the encoder sequence and the decoder sequence; and
generating, using the trained model, a prediction based on an input data set.
2. The computer-implemented method of claim 1 , wherein training the model using the encoder sequence and the decoder sequence comprises:
training the encoder using the encoder sequence of the encoding; and
training the decoder using the decoder sequence of the encoding.
3. The computer-implemented method of claim 2 , wherein training the model using the encoder sequence and the decoder sequence further comprises:
prior to training the decoder using the decoder sequence of the encoding:
prepending a start of sequence token to the decoder sequence of the encoding; and
appending an end of sequence token to the decoder sequence of the encoding.
4. The computer-implemented method of claim 1 , wherein generating the prediction comprises:
generating an input encoding of the input data set;
generating an output sequence comprising a start of output sequence token, and one or more output sequence tokens; and
generating the prediction based on the output sequence.
5. The computer-implemented method of claim 4 , wherein the one or more output sequence tokens are generated by:
generating a next output sequence token by providing the input encoding to the trained model; and
appending the next output sequence token to the output sequence until the next output sequence token comprises an end of output sequence token.
6. The computer-implemented method of claim 1 , wherein each encoding comprises an attention weight for each token in the encoding, and
wherein the attention weights are based on a feed-forward analysis using a position of each token in the encoding.
7. The computer-implemented method of claim 1 , wherein training the model comprises training the encoder by updating an attention weight for at least one token in the encoder sequence of the encoding.
8. The computer-implemented method of claim 1 , wherein training the model comprises training the decoder by updating an attention weight for at least one token in the decoder sequence of the encoding to an attention weight associated with at least one token in the encoder sequence of the encoding.
9. The computer-implemented method of claim 1 , wherein an encoding of a sequence comprises a vector representation the sequence.
10. The computer-implemented method of claim 1 , wherein encoder sequences of the plurality of training sequences comprise a set of dialog prompts.
11. The computer-implemented method of claim 1 , wherein decoder sequences of the plurality of training sequences comprise a set of dialog responses.
12. The computer-implemented method of claim 1 , wherein:
the plurality of training sequences comprises a vocabulary; and
an encoding for a sequence comprises one hundred percent coverage for the vocabulary.
13. A device, comprising:
a processor; and
a memory in communication with the processor and storing instructions that, when read by the processor, cause the device to:
initialize a model having a sequence to sequence network architecture comprising an encoder and a decoder;
train the model using a plurality of training sequences, wherein each training sequence comprises an encoder sequence and a decoder sequence, and wherein the instructions, when executed by the processor, cause the device to train the model by:
generating an encoding of each training sequence of the plurality of training sequences;
for each encoding:
randomly inserting an informative padding, comprising a random sampling of encoded tokens from the plurality of training sequences, into the encoder sequence of the encoding; and
training the model using the encoder sequence and the decoder sequence; and
generate, using the trained model, a prediction based on an input data set.
14. The device of claim 13 , wherein the instructions, when executed by the processor, further cause the device to train the model by:
training the encoder using the encoder sequence of the encoding; and
training the decoder using the decoder sequence of the encoding.
15. The device of claim 14 , wherein the instructions, when executed by the processor, further cause the device to train the model using the encoder sequence and the decoder sequence by:
prior to training the decoder using the decoder sequence of the encoding:
prepending a start of sequence token to the decoder sequence of the encoding; and
appending an end of sequence token to the decoder sequence of the encoding.
16. The device of claim 13 , wherein each encoding comprises an attention weight for each token in the encoding, and
wherein the attention weights are based on a feed-forward analysis using a position of each token in the encoding.
17. The device of claim 13 , wherein the instructions, when executed by the processor, further cause the device to train the encoder by:
updating an attention weight for at least one token in the encoder sequence of the encoding; and
updating an attention weight for at least one token in the decoder sequence of the encoding to the attention weight associated with at least one token in the encoder sequence of the encoding.
18. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
initializing a model having a sequence to sequence network architecture comprising an encoder and a decoder;
training the model using a plurality of training sequences, wherein each training sequence comprises an encoder sequence and a decoder sequence, and wherein training the model comprises:
generating an encoding of each training sequence of the plurality of training sequences;
for each encoding:
randomly inserting an informative padding, comprising a random sampling of encoded tokens from the plurality of training sequences, into the encoder sequence of the encoding; and
training the encoder using the encoder sequence and the decoder sequence; and
generating, using the trained model, a prediction based on an input data set.
19. The non-transitory machine-readable medium of claim 18 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform steps comprising:
train the model using the plurality of training sequences by:
training the encoder using the encoder sequence of the encoding; and
training the decoder using the decoder sequence of the encoding.
20. The non-transitory machine-readable medium of claim 19 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform steps comprising:
prior to training the decoder using the decoder sequence of the encoding:
prepending a start of sequence token to the decoder sequence of the encoding; and
appending an end of sequence token to the decoder sequence of the encoding.