Multilingual grammatical error correction
A method of training a text-generating model for grammatical error correction (GEC) includes obtaining a multilingual set of text samples where each text sample includes a monolingual textual representation of a respective sentence. The operations also include, for each text sample of the multilingual set of text samples, generating a corrupted synthetic version of the respective text sample where the corrupted synthetic version of the respective text sample includes a grammatical change to the monolingual textual representation of the respective sentence associated with the respective text sample. The operations further include training the text-generating model using a training set of sample pairs. Each sample pair in the training set of sample pairs includes one of the respective text samples of the multilingual set of text samples and the corresponding corrupted synthetic version of the one of the respective text samples of the multilingual set of text samples.
1 . A computer-implemented method of training a text-generating model for grammatical error correction (GEC), the method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a multilingual set of grammatically correct text samples, each grammatically correct text sample comprising a monolingual textual representation of a respective sentence;
for each grammatically correct text sample of the multilingual set of grammatically correct text samples, generating a corrupted synthetic ungrammatical text version of the respective grammatically correct text sample by making a grammatical change to the monolingual textual representation of the respective sentence associated with the respective grammatically correct text sample, wherein the grammatical change is made to the monolingual textual representation of the respective sentence associated with the respective grammatically correct text sample without using a dictionary that is specific to a language of the monolingual textual representation of the respective segment such that the grammatical change made to the monolingual textual representation of the respective sentence is agnostic to the language of the monolingual textual representation of the respective sentence; and
training the text-generating model using a training set of sample pairs, each sample pair in the training set of sample pairs comprising:
one of the respective grammatically correct text samples of the multilingual set of grammatically correct text samples; and
the corresponding corrupted synthetic ungrammatical text version of the one of the respective grammatically correct text samples of the multilingual set of grammatically correct text samples.
2 . The method of claim 1 , wherein the operations further comprise, after training the text-generating model, fine-tuning the trained text-generating model using supervised training data, the supervised training data comprising non-synthetic text pairs, each non-synthetic text pair comprising an ungrammatical text sample and a grammatical text version of the ungrammatical text sample.
3 . The method of claim 1 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises removing more than one character from the respective sentence associated with the respective grammatically correct text sample.
4 . The method of claim 1 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises replacing a first set of characters from the respective sentence associated with the respective grammatically correct text sample with a second set of characters different from the first set of characters.
5 . The method of claim 1 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises inserting one or more characters into the respective sentence associated with the respective grammatically correct text sample.
6 . The method of claim 1 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises changing a character-case for a character of a word of the respective sentence associated with the respective grammatically correct text sample.
7 . The method of claim 1 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises randomly applying a corruption operation to the respective sentence associated with the respective grammatically correct text sample, the corruption operation comprising at least one of:
removing more than one characters from the respective sentence associated with the respective grammatically correct text sample;
replacing a first set of characters from the respective sentence associated with the respective grammatically correct text sample with a second set of characters different from the first set of characters;
inserting one or more characters into the respective sentence associated with the respective grammatically correct text sample; or
changing a character-case of a word of the respective sentence associated with the respective grammatically correct text sample,
wherein each corrupted synthetic ungrammatical text version is unique with respect to the other corrupted synthetic ungrammatical text versions of the text samples.
8 . The method of claim 1 , wherein the text-generating model comprises a transformer encoder-decoder architecture.
9 . The method of claim 1 , wherein the operations further comprise pre-training the text-generating model with a multilingual training corpus based on a masked-language objective.
10 . The method of claim 1 , wherein after training the text-generating model for GEC, the trained text-generating model is configured to:
receive, as input, a first input text in a first language that includes grammatical errors and generate, as output, a first output text in the first language that corrects the grammatical errors; and
receiving, as input, a second input text in a different second language that includes grammatical errors and generate, as output from the trained text-generating model, a second output text in the second language that corrects the grammatical errors.
11 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining a multilingual set of grammatically correct text samples, each grammatically correct text sample comprising a monolingual textual representation of a respective sentence;
for each grammatically correct text sample of the multilingual set of grammatically correct text samples, generating a corrupted synthetic ungrammatical text version of the respective grammatically correct text sample by making a grammatical change to the monolingual textual representation of the respective sentence associated with the respective grammatically correct text sample, wherein the grammatical change is made to the monolingual textual representation of the respective sentence associated with the respective grammatically correct text sample without using a dictionary that is specific to a language of the monolingual textual representation of the respective segment such that the grammatical change made to the monolingual textual representation of the respective sentence is agnostic to the language of the monolingual textual representation of the respective sentence; and
training the text-generating model using a training set of sample pairs, each sample pair in the training set of sample pairs comprising:
one of the respective grammatically correct text samples of the multilingual set of grammatically correct text samples; and
the corresponding corrupted synthetic ungrammatical text version of the one of the respective grammatically correct text samples of the multilingual set of grammatically correct text samples.
12 . The system of claim 11 , wherein the operations further comprise, after training the text-generating model, fine-tuning the trained text-generating model using supervised training data, the supervised training data comprising non-synthetic text pairs, each non-synthetic text pair comprising an ungrammatical text sample and a grammatical text version of the ungrammatical text sample.
13 . The system of claim 11 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises removing more than one characters from the respective sentence associated with the respective grammatically correct text sample.
14 . The system of claim 11 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises replacing a first set of characters from the respective sentence associated with the respective grammatically correct text sample with a second set of characters different from the first set of characters.
15 . The system of claim 11 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises inserting one or more characters into the respective sentence associated with the respective grammatically correct text sample.
16 . The system of claim 11 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises changing a character-case for a character of a word of the respective sentence associated with the respective grammatically correct text sample.
17 . The system of claim 11 , wherein generating the corrupted synthetic ungrammatical text version of the respective grammatically correct text sample comprises randomly applying a corruption operation to the respective sentence associated with the respective grammatically correct text sample, the corruption operation comprising at least one of:
removing more than one characters from the respective sentence associated with the respective grammatically correct text sample;
replacing a first set of characters from the respective sentence associated with the respective grammatically correct text sample with a second set of characters different from the first set of characters;
inserting one or more characters into the respective sentence associated with the respective grammatically correct text sample; or
changing a character-case of a word of the respective sentence associated with the respective grammatically correct text sample,
wherein each corrupted synthetic ungrammatical text version is unique with respect to the other corrupted synthetic ungrammatical text versions of the text samples.
18 . The system of claim 11 , wherein the text-generating model comprises a transformer encoder-decoder architecture.
19 . The system of claim 11 , wherein the operations further comprise pre-training the text-generating model with a multilingual training corpus based on a masked-language objective.
20 . The system of claim 11 , wherein after training the text-generating model for GEC, the trained text-generating model is configured to:
receive, as input, a first input text in a first language that includes grammatical errors and generate, as output, a first output text in the first language that corrects the grammatical errors; and
receiving, as input, a second input text in a different second language that includes grammatical errors and generate, as output from the trained text-generating model, a second output text in the second language that corrects the grammatical errors.