Techniques and models for multilingual text rewriting
The technology provides a model-based approach for multilingual text rewriting that is applicable across many languages and across different styles including formality levels or other textual attributes. The model is configured to manipulate both language and textual attributes jointly. This approach supports zero-shot formality-sensitive translation, with no labeled data in the target language. An encoder-decoder architectural approach with attribute extraction is used to train rewriter models that can thus be used in “universal” textual rewriting across many different languages. A cross-lingual learning signal can be incorporated into the training approach. Certain training processes do not employ any exemplars. This approach enables not just straight translation, but also the ability to create new sentences with different attributes.
1. A system configured for a multilingual text rewriting model, the system comprising:
memory configured to store a set of text exemplars in a source language and a set of rewritten texts in a plurality of languages different from the source language; and
one or more processing elements operatively coupled to the memory, the one or more processing elements implementing a multilingual text rewriter as a neural network having:
a corruption module configured to generate a corrupted version of an input text sequence obtained based on the set of text exemplars in the source language stored in the memory, the corruption module employing a corruption function that uses negation of an attribute vector associated with the input text sequence;
an encoder module comprising an encoder neural network configured to receive the corrupted version of the input text sequence from the corruption module and to generate a set of encoded representations of the corrupted version of the input text sequence;
a style extractor module configured to extract a set of style vector representations associated with the input text sequence; and
a decoder module comprising a decoder neural network configured to receive the set of encoded representations of the corrupted version of the input text sequence and the set of style vector representations and to output the set of rewritten texts in the plurality of languages, each style vector representation of the set being added element-wise to one of the set of encoded representations of the corrupted version of the input text sequence;
wherein a set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set; and
wherein the system is configured to provide rewritten texts in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence.
2. The system of claim 1 , wherein the encoder module, the style extractor module and the decoder module are configured as transformer stacks initialized from a common pretrained language model.
3. The system of claim 1 , wherein both the encoder module and the decoder module are attention-based neural network modules.
4. The system of claim 1 , wherein the corruption function is a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the multilingual text rewriting model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is the attribute vector of span s1.
5. The system of claim 4 , wherein the corruption function C is as follows:
C:=f (⋅,source language,− Vs 1).
6. The system of claim 1 , wherein the style extractor module includes a set of style extractor elements including a first subset configured to operate on different style exemplars and a second subset configured to operate on the input text sequence prior to corruption by the corruption module.
7. The system of claim 1 , wherein during training the corruption module is configured to employ at least one of token-level corruption or style-aware back-translation corruption.
8. The system of claim 1 , wherein the set of model weights is initialized with the weights of a pretrained text-to-text model.
9. The system of claim 1 , wherein the model weights for the style extractor module are not tied to the weights of the encoder module during training.
10. The system of claim 1 , wherein the system is configured to extract pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar, and configured to use a first-occurring span in the text as an exemplar of attributes of a second span in the text.
11. The system of claim 1 , wherein a cross-lingual learning signal is added to a training objective for training the model.
12. The system of claim 1 , wherein the encoder module is configured to use negation of a true exemplar vector associated with the input text sequence as the attribute vector in a forward pass operation.
13. The system of claim 1 , wherein the decoder module is configured to receive a set of stochastic tuning ranges that provide conditioning for the decoder module.
14. The system of claim 1 , wherein:
the set of style vector representations corresponds to a set of attributes associated with the input text sequence; and
the system is configured to input a set of exemplars illustrating defined attributes and to extract corresponding attribute vectors for use in rewriting the input text sequence into the plurality of languages according to the defined attributes.
15. The system of claim 14 , wherein:
the system is configured to form an attribute delta vector including a scale factor; and
the attribute delta vector is added to the set of encoded representations of the corrupted version of the input text sequence before processing by the decoder module.
16. A computer-implemented method for providing multilingual text rewriting according to a machine learning model, the method comprising:
obtaining an input text sequence based on a set of text exemplars in a source language stored in a memory;
generating, by a corruption module, a corrupted version of the input text sequence, the corruption module employing a corruption function that uses negation of an attribute vector associated with the input text sequence;
receiving, by an encoder neural network of an encoder module, the corrupted version of the input text sequence from the corruption module;
generating, by the encoder module, a set of encoded representations of the corrupted version of the input text sequence;
extracting, by a style extractor module, a set of style vector representations associated with the input text sequence;
receiving, by a decoder neural network of a decoder module, the set of encoded representations of the corrupted version of the input text sequence and the set of style vector representations, in which each style vector representation of the set is added element-wise to one of the set of encoded representations of the corrupted version of the input text sequence;
outputting, by the decoder module, a set of rewritten texts in a plurality of languages different from the source language; and
storing the rewritten texts in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence;
wherein a set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set.
17. The method of claim 16 , wherein the corruption function is a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the machine learning model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is the attribute vector of span s1.
18. The method of claim 16 , further comprising:
extracting pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar; and
using a first-occurring span in the text as an exemplar of attributes of a second span in the text.
19. The method of claim 16 , further comprising adding a cross-lingual learning signal to a training objective for training the model.
20. The method of claim 16 , further comprising applying a set of stochastic tuning ranges to selectively condition the decoder module.
21. The method of claim 16 , wherein:
the set of style vector representations corresponds to a set of attributes associated with the input text sequence; and
outputting the set of rewritten texts includes generating one or more versions of the input text sequence in selected ones of the plurality of languages according to the set of attributes.
22. A non-transitory computer-readable medium storing instructions which, when executed, cause a computing device to perform a method of providing multilingual text rewriting according to a machine learning model, the method comprising:
obtaining an input text sequence based on a set of text exemplars in a source language stored in a memory;
generating, by a corruption module, a corrupted version of the input text sequence, the corruption module employing a corruption function that uses negation of an attribute vector associated with the input text sequence;
receiving, by an encoder neural network of an encoder module, the corrupted version of the input text sequence from the corruption module;
generating, by the encoder module, a set of encoded representations of the corrupted version of the input text sequence;
extracting, by a style extractor module, a set of style vector representations associated with the input text sequence;
receiving, by a decoder neural network of a decoder module, the set of encoded representations of the corrupted version of the input text sequence and the set of style vector representations, in which each style vector representation of the set is added element-wise to one of the set of encoded representations of the corrupted version of the input text sequence;
outputting, by the decoder module, a set of rewritten texts in a plurality of languages different from the source language; and
storing the rewritten texts in selected ones of the plurality of languages according to a change in a least a sentiment or a formality of the input text sequence;
wherein a set of model weights is shared by the encoder module and the style extractor module, and a unique token is appended to the input text sequence for style extraction instead of mean-pooling all of the style vector representations in the set.
23. The non-transitory computer-readable medium of claim 22 , wherein the corruption function is a corruption function C for a given pair of non-overlapping spans (s1, s2) of the input text sequence, so that the machine learning model is capable of reconstructing span s2 from C(s2) and Vs1, where Vs1 is the attribute vector of span s1.
24. The non-transitory computer-readable medium of claim 22 , wherein the method further comprises:
extracting pairs of random non-overlapping spans of tokens from each line of text in a given text exemplar; and
using a first-occurring span in the text as an exemplar of attributes of a second span in the text.
25. The non-transitory computer-readable medium of claim 22 , wherein the method further comprises adding a cross-lingual learning signal to a training objective for training the model.