Semi-autoregressive text editing
Provided are improved machine learning-based text editing models. Specifically, example implementations include a flexible semi-auto-regressive text-editing approach for generation, designed to derive the maximum benefit from non-auto-regressive text-editing and autoregressive decoding. In contrast to conventional sequence-to-sequence (seq2seq) models, the proposed approach is fast at inference time, while being capable of modeling flexible input-output transformations.
1 . A computer system that performs text editing, the computer system comprising:
one or more processors;
a machine-learned text editing model configured to receive and process a source text string to generate an output text string, the output text string comprising an edited version of the source text string, the machine-learned text editing model comprising:
an encoder model configured to process the source text string in a non-autoregressive manner to generate an intermediate text representation, wherein the source text string comprises a plurality of source tokens, and wherein the intermediate text representation indicates: (1) a subset of the plurality of source tokens to be maintained for the output text string and (2) an ordering of the subset of the plurality of source tokens to be maintained for the output text string; and
a decoder model configured to process the intermediate text representation in an autoregressive manner to select one or more additional tokens to insert into the subset of the plurality of source tokens to generate the output text string, wherein the decoder model is configured to first predict a position token indicating a position at which one or more of the additional tokens should be inserted into the intermediate text representation and then to second predict the one or more of the additional tokens to be inserted into the intermediate text representation at the position;
wherein the encoder model is configured to process the source text string in the non-autoregressive manner to generate the intermediate text representation without indicating the position at which one or more of the additional tokens should be inserted into the intermediate text representation; and
one or more non-transitory computer readable media that store instructions that, when executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
obtaining the source text string;
processing the source text string with the machine-learned text editing model to generate the output text string; and
providing the output text string as an output.
2 . The computer system of claim 1 , wherein the encoder model comprises:
a text embedding model configured to process the source text string to generate a hidden representation;
a tagging model configured to assign a respective tag to each of the source tokens in the source text string, the respective tag for each source token indicating whether or not such source token is included in the subset of the plurality of source tokens to be maintained for the output text string; and
a pointer model configured to generate the ordering of the subset of the plurality of source tokens to be maintained for the output text string based at least in part on the hidden representation and the respective tag assigned to each of the source tokens.
3 . The computer system of claim 1 , wherein each of the encoder model and the decoder model comprises a transformer neural network.
4 . The computer system of claim 3 , wherein at least the transformer neural network in the decoder model comprises a T5 pre-trained transformer neural network that has been pre-trained to insert missing spans.
5 . The computer system of claim 1 , wherein the decoder model has been pre-trained with a denoising objective.
6 . The computer system of claim 1 , wherein the encoder model comprises one or more Sinkhorn layers that normalize over both rows and columns of an intra-pointer attention.
7 . The computer system of claim 1 , wherein the decoder model being configured to first predict the position token comprises repurposing one or more special span_i tokens to indicate the position token indicating the position at which one or more of the additional tokens should be inserted into the intermediate text representation.
8 . A computer-implemented method to train a text editing model, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a training example comprising a source text string and target text string;
processing, by the computing system, the source text string with the text editing model to generate an output text string, wherein processing the source text string with the text editing model to generate the output text string comprises:
processing, by the computing system, the source text string in a non-autoregressive manner with an encoder model of the text editing model to generate an intermediate text representation, wherein the source text string comprises a plurality of source tokens, and wherein the intermediate text representation indicates: (1) a subset of the plurality of source tokens to be maintained for the output text string and (2) an ordering of the subset of the plurality of source tokens to be maintained for the output text string; and
processing, by the computing system, the intermediate text representation in an autoregressive manner with a decoder model of the text editing model to select one or more additional tokens to insert into the subset of the plurality of source tokens to generate the output text string;
wherein the encoder model is configured to process the source text string in the non-autoregressive manner to generate the intermediate text representation without indicating a position at which one or more of the additional tokens should be inserted into the intermediate text representation;
evaluating, by the computing system, a combined loss function that respectively compares (i) each of: a set of ground truth labels for the source tokens to be maintained, a ground truth ordering, and the target text string respectively to (ii) each of: the subset of the plurality of source tokens, the ordering of the subset of the plurality of source tokens, and the output text string; and
modifying, by the computing system, one or more parameters of the text editing model based on the combined loss function.
9 . The computer-implemented method of claim 8 , wherein the encoder model comprises:
a text embedding model configured to process the source text string to generate a hidden representation;
a tagging model configured to assign a respective tag to each of the source tokens in the source text string, the respective tag for each source token indicating whether or not such source token is included in the subset of the plurality of source tokens to be maintained for the output text string; and
a pointer model configured to generate the ordering of the subset of the plurality of source tokens to be maintained for the output text string based at least in part on the hidden representation and the respective tag assigned to each of the source tokens.
10 . The computer-implemented method of claim 8 , wherein the combined loss function comprises a tagging loss term that evaluates a probability of the encoder model outputting the set of ground truth labels for the source tokens to be maintained for the output text string.
11 . The computer-implemented method of claim 8 , wherein the combined loss function comprises a pointing loss term that compares the ordering with the ground truth ordering.
12 . The computer-implemented method of claim 8 , wherein the combined loss function comprises an insertion loss term that evaluates a probability of the decoder model outputting a set of ground truth tokens to be included in the output text string.
13 . The computer-implemented method of claim 8 , wherein each of the encoder model and the decoder model comprises a transformer neural network.
14 . The computer-implemented method of claim 8 , wherein the encoder model comprises one or more Sinkhorn layers that normalize over both rows and columns of an intra-pointer attention.
15 . One or more non-transitory computer readable media that store:
a machine-learned text editing model configured to receive and process a source text string to generate an output text string, the output text string comprising an edited version of the source text string, the machine-learned text editing model comprising:
an encoder model configured to process the source text string in a non-autoregressive manner to generate an intermediate text representation, wherein the source text string comprises a plurality of source tokens, and wherein the intermediate text representation indicates: (1) a subset of the plurality of source tokens to be maintained for the output text string and (2) an ordering of the subset of the plurality of source tokens to be maintained for the output text string; and
a decoder model configured to process the intermediate text representation in an autoregressive manner to first predict a position token indicating a position at which one or more additional tokens should be inserted into the intermediate text representation and then to second select the one or more additional tokens to insert into the subset of the plurality of source tokens at the position to generate the output text string;
wherein the encoder model is configured to process the source text string in the non-autoregressive manner to generate the intermediate text representation without indicating the position at which one or more of the additional tokens should be inserted into the intermediate text representation.
16 . The one or more non-transitory computer readable media of claim 15 , wherein the encoder model comprises:
a text embedding model configured to process the source text string to generate a hidden representation;
a tagging model configured to assign a respective tag to each of the source tokens in the source text string, the respective tag for each source token indicating whether or not such source token is included in the subset of the plurality of source tokens to be maintained for the output text string; and
a pointer model configured to generate the ordering of the subset of the plurality of source tokens to be maintained for the output text string based at least in part on the hidden representation and the respective tag assigned to each of the source tokens.
17 . The one or more non-transitory computer readable media of claim 15 , wherein each of the encoder model and the decoder model comprises a transformer neural network.
18 . The one or more non-transitory computer readable media of claim 17 , wherein at least the transformer neural network in the decoder model comprises a T5 pre-trained transformer neural network that has been pre-trained to insert missing spans.
19 . The one or more non-transitory computer readable media of claim 18 , wherein the decoder model comprises a single T5 decoder transformer layer.
20 . The one or more non-transitory computer-readable media of claim 15 , wherein the encoder model comprises one or more Sinkhorn layers that normalize over both rows and columns of an intra-pointer attention.