Automatically generating annotated ground-truth corpus for training NLU model

1 . A method comprising:

accessing an initial corpus of ground-truth utterances in a first language, each ground truth utterance annotated with one or more intent labels and one or more slot labels, each slot label having an associated slot value;

generating a translated, annotated corpus in a second language by, for each annotated utterance in the initial corpus:

delexicalizing the annotated utterance by replacing each slot value with its respective slot label in the annotated ground-truth utterance, wherein each delexicalized utterance comprises a first portion including one or more slot labels without any slot values and a second portion including all other words in the delexicalized utterance;

creating a translated delexicalized utterance comprising the second portion in a second language and the first portion in the first language by translating each second portion of the delexicalized utterance into a second language without modifying the first portion of the delexicalized utterance;

creating a translated slots portion comprising (1) each slot value in the second language and (2) its corresponding slot label in the first language by translating, separately from the translation of each portion of the delexicalized utterance, each slot value to the second language;

creating a translated annotated utterance by adding, to the translated delexicalized utterance, each translated slot value from the translated slots portion at that slot value's corresponding slot label's position in the delexicalized utterance; and

creating a multilingual natural language model by training a natural-language model using the translated corpus.

2 . The method of claim 1 , further comprising generating, prior to creating the multilingual natural language model, an augmented translated corpus by:

randomly selecting, from the translated, annotated corpus, a plurality of annotated utterances;

for each selected annotated utterance, delexicalizing the utterance by replacing the slot values in the utterance with the corresponding slot labels; and

adding each delexicalized utterance to the translated, annotated corpus.

3 . The method of claim 1 , further comprising generating, prior to creating the multilingual natural language model, an augmented translated corpus by:

randomly selecting, from the translated, annotated corpus, a plurality of annotated utterances;

for each selected annotated utterance, codeswitching the utterance by:

selecting a word in the utterance;

translating the selected word into a different language;

replacing the selected word with the translated word; and

adding each codeswitched utterance to the translated, annotated corpus.

4 . The method of claim 1 , further comprising:

generating a plurality of translated, annotated corpuses in a plurality of corresponding second languages; and

training the natural-language model using each of the plurality of translated corpuses.

5 . The method of claim 1 , wherein the multilingual natural language model comprises a plurality of classification heads.

6 . The method of claim 5 , wherein at least one of the classification heads is a bag-of-slots classification head or a language-order prediction head.

7 . The method of claim 1 , wherein the multilingual natural language model is part of a dialogue system.

8 . A method comprising:

receiving an input utterance; and

determining, using a multilingual natural language model, one or more intent labels and one or more slot labels for the utterance, wherein the multilingual natural language model was generated by:

accessing an initial corpus of ground-truth utterances in a first language, each ground truth utterance annotated with one or more intent labels and one or more slot labels, each slot label having an associated slot value;

generating a translated, annotated corpus in a second language by, for each annotated utterance in the initial corpus:

delexicalizing the annotated utterance by replacing each slot value with its respective slot label in the annotated ground-truth utterance, wherein each delexicalized utterance comprises a first portion including one or more slot labels without any slot values and a second portion including all other words in the delexicalized utterance;

creating a translated delexicalized utterance comprising the second portion in a second language and the first portion in the first language by translating each second portion of the delexicalized utterance into a second language without modifying the first portion of the delexicalized utterance;

creating a translated slots portion comprising (1) each slot value in the second language and (2) its corresponding slot label in the first language by translating, separately from the translation of each portion of the delexicalized utterance, each slot value to the second language;

creating a translated annotated utterance by adding, to the translated delexicalized utterance, each translated slot value from the translated slots portion at that slot value's corresponding slot label's position in the delexicalized utterance; and

creating the multilingual natural language model by training a natural-language model using the translated corpus.

9 . The method of claim 8 , wherein the multilingual natural language model was further generated by an augmented translated corpus, wherein the augmented translated corpus was generated by:

randomly selecting, from the translated, annotated corpus, a plurality of annotated utterances;

for each selected annotated utterance, delexicalizing the utterance by replacing the slot values in the utterance with the corresponding slot labels; and

adding each delexicalized utterance to the translated, annotated corpus.

10 . The method of claim 8 , wherein the multilingual natural language model was further generated by an augmented translated corpus, wherein the augmented translated corpus was generated by:

randomly selecting, from the translated, annotated corpus, a plurality of annotated utterances;

for each selected annotated utterance, codeswitching the utterance by:

selecting a word in the utterance;

translating the selected word into a different language;

replacing the selected word with the translated word; and

adding each codeswitched utterance to the translated, annotated corpus.

11 . The method of claim 8 , wherein the multilingual natural language model was further generated by:

generating a plurality of translated, annotated corpuses in a plurality of corresponding second languages; and

training the natural-language model using each of the plurality of translated corpuses.

12 . The method of claim 8 , wherein determining, using a multilingual natural language model, one or more intent labels and one or more slot labels for the utterance comprises:

translating the utterance into one or more parallel utterances, each parallel utterance being in a different language;

determining, by the multilingual natural language model and for each utterance, one or more initial intent labels and one or more initial slot labels; and

determining, based on the initial intent labels and the one or more initial slot labels, one or more final intent labels and one or more final slot labels for the utterance.

13 . The method of claim 12 , wherein the one or more final intent labels comprise a most frequently determined one or more initial intent labels, and the one or more final slot labels comprise a most frequently determined one or more initial slot labels.

14 . The method of claim 8 , wherein the multilingual natural language model is part of a dialogue system.

15 . One or more non-transitory computer readable storage media storing software comprising:

a multilingual natural language model defined by a plurality of model parameters, the plurality of model parameters generated by:

accessing an initial corpus of ground-truth utterances in a first language, each ground truth utterance annotated with one or more intent labels and one or more slot labels, each slot label having an associated slot value;

generating a translated, annotated corpus in a second language by, for each annotated utterance in the initial corpus:

delexicalizing the annotated utterance by replacing each slot value with its respective slot label in the annotated ground-truth utterance, wherein each delexicalized utterance comprises a first portion including one or more slot labels without any slot values and a second portion including all other words in the delexicalized utterance;

creating a translated delexicalized utterance comprising the second portion in a second language and the first portion in the first language by translating each second portion of the delexicalized utterance into a second language without modifying the first portion of the delexicalized utterance;

creating a translated slots portion comprising (1) each slot value in the second language and (2) its corresponding slot label in the first language by translating, separately from the translation of each portion of the delexicalized utterance, each slot value to the second language;

creating a translated annotated utterance by adding, to the translated delexicalized utterance, each translated slot value from the translated slots portion at that slot value's corresponding slot label's position in the delexicalized utterance; and

training a natural-language model using the translated corpus.

16 . The media of claim 15 , wherein at least some of the plurality of model parameters are further determined by, prior to training:

randomly selecting, from the translated, annotated corpus, a plurality of annotated utterances;

for each selected annotated utterance, delexicalizing the utterance by replacing the slot values in the utterance with the corresponding slot labels; and

adding each delexicalized utterance to the translated, annotated corpus.

17 . The media of claim 15 , wherein at least some of the plurality of model parameters are further determined by, prior to training:

randomly selecting, from the translated, annotated corpus, a plurality of annotated utterances;

for each selected annotated utterance, codeswitching the utterance by:

selecting a word in the utterance;

translating the selected word into a different language;

replacing the selected word with the translated word; and

adding each codeswitched utterance to the translated, annotated corpus.

18 . The media of claim 15 , wherein at least some of the plurality of model parameters are further determined by:

generating a plurality of translated, annotated corpuses in a plurality of corresponding second languages; and

training the natural-language model using each of the plurality of translated corpuses.

19 . The media of claim 15 , wherein the media is coupled to one or more processors that are operable to execute the software to determine, using the multilingual natural language model, one or more intent labels and one or more slot labels for an input utterance.

20 . The media of claim 19 , wherein the media and the one or more processors are part of a dialogue system.