IP Library › Granted Patent US 12,380,288
Granted Patent B2
US 12,380,288 · App. 17/931,002 · Granted Aug 5, 2025

Multilingual unsupervised neural machine translation with denoising adapters

Inventors: Alexandre Bérard (Grenoble, FR); Laurent Besacier (Seyssinet Pariset, FR); Matthias Gallé (Eybens, FR); Ahmet Üstün (Groningen, NL)
Assignee: NAVER CORPORATION
G06F40/58G06F40/126G06F40/47G06N3/088
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,380,288
App. No.
17/931,002
Granted
Aug 5, 2025
Kind
B2
Abstract

Methods and systems for unsupervised training for a neural multilingual sequence-to-sequence (seq2seq) model. Denoising adapters for each of one or more languages is inserted into an encoder and/or a decoder of the seq2seq model. Parameters of the one or more denoising adapters are trained on a language-specific denoising task using monolingual text for each of the one or more languages. Cross-attention weights of the seq2seq model with the trained denoising adapter layers are fine-tuned on a translation task in at least one of the one or more languages with parallel data.

Claims (61)

1. A method for translating a sequence, comprising:

translating, by one or more processors, a new sequence in one or more languages using a neural multilingual sequence-to-sequence (seq2seq) model, wherein the seq2seq model is trained using an unsupervised training method comprising:

inserting, in a memory accessed by one or more processors, one or more denoising adapters for each of one or more languages into an encoder and/or a decoder of the seq2seq model;

training, by one or more processors, parameters of each of the one or more denoising adapters on a language-specific denoising task using monolingual text for each of the one or more languages; and

fine-tuning, by one or more processors, cross-attention weights of the seq2seq model with the denoising adapter layers and the trained parameters on a translation task in at least one of the one or more languages with parallel data.

2. The method of claim 1 , wherein the seq2seq model comprises a machine translation model.

3. The method of claim 1 , wherein the seq2seq model comprises a transformer encoder and a transformer decoder, wherein the transformer decoder comprises a cross-attention layer;

wherein said fine-tuning comprises fine-tuning cross-attention weights of the cross-attention layer.

4. The method of claim 1 , wherein each of the denoising adapters comprises lightweight residual layers.

5. The method of claim 1 , wherein each of the denoising adapters comprises a normalization layer, a down projection layer, a non-linear activation function, an up-projection layer, and a residual connection combining an output of the up-projection layer with an input to the adapter.

6. The method of claim 1 , wherein the seq2seq model is pretrained for translating from a sequence in a source language to a sequence in a target language, wherein the source and target languages are taken from a set of languages, wherein the seq2seq model has a vocabulary of tokens in each of the set of languages.

7. The method of claim 6 , wherein for at least one language in the set of languages, only monolingual data is used for the language when training the seq2seq model for translating to or from a sequence in that language.

8. The method of claim 1 , wherein said inserting the one or more denoising adapters comprises inserting one or more denoising adapters for each of the one or more languages into each of the encoder and the decoder of the seq2seq model.

9. The method of claim 1 , wherein the encoder comprises a transformer encoder and the decoder comprises a transformer decoder, wherein the transformer decoder comprises a cross-attention layer, wherein said fine-tuning cross-attention weights learns the cross-attention layer;

wherein the transformer encoder layer comprises a feedforward encoder layer and a self-attention layer having a plurality of attention heads;

wherein the transformer decoder comprises a feedforward decoder layer, a self-attention layer having a plurality of attention heads, and the cross-attention layer;

wherein said inserting the one or more denoising adapters comprises inserting one or more denoising adapter layers for each of the one or more languages into each of the transformer encoder and the transformer decoder following self-attention and feedforward layers;

wherein the output of the transformer encoder is fed to the cross-attention layer of the transformer decoder.

10. The method of claim 1 , wherein said training parameters of the one or more denoising adapters for each language uses only monolingual texts for each of the one or more languages.

11. The method of claim 1 , wherein the denoising adapters learn language-specific encoding and decoding in each of the one or more languages by denoising the monolingual texts in each respective language.

12. The method of claim 11 , wherein the denoising comprises performing a noise function on the monolingual text, the noise function comprising one or more of: token masking, token deletion, word-span masking, sentence permutation, or document rotation.

13. The method of claim 1 , wherein said training parameters of each of the one or more denoising adapters is performed while freezing other parameters of the seq2seq model.

14. The method of claim 1 , wherein said training parameters of each of the one or more denoising adapters is performed while freezing all parameters of the seq2seq model other than the parameters of the one or more denoising adapters.

15. The method of claim 1 , wherein said training parameters of each of the one or more denoising adapters is performed while freezing all parameters of the seq2seq model other than the parameters of the one or more denoising adapters and weights of an output projection layer in the seq2seq model.

16. The method of claim 1 , wherein the seq2seq model is pretrained for translating from a sequence in a source language to a sequence in a target language, wherein the source and target languages are taken from a set of languages for which the seq2seq model is pretrained, and wherein one or more of the one or more languages is not in the set of languages.

17. The method of claim 1 , wherein the parallel data used for said fine-tuning cross-attention weights comprises parallel translations pairing a language with a common language.

18. The method of claim 1 , wherein said fine-tuning finetunes cross-attention weights of the seq2seq model on the translation task while the trained denoising adapters for all of the one or more languages are inserted in the encoder and/or decoder.

19. The method of claim 18 , wherein said fine-tuning cross-attention weights is performed while freezing parameters of the inserted trained denoising adapters and parameters of the seq2seq model other than the cross-attention weights.

20. The method of claim 1 , wherein the fine-tuned cross-attention weights are shared among all of the one or more languages.

21. The method of claim 1 , wherein said fine-tuning is for a translation task.

22. The method of claim 1 , further comprising:

further fine-tuning the seq2seq model using back translation for the one or more languages.

23. The method of claim 1 , wherein the seq2seq model is not trained using back-translation.

24. The method of claim 1 , wherein the translation is supervised.

25. The method of claim 1 , wherein the translation is unsupervised.

26. A method for translating a sequence comprising:

translating, by one or more processors, a new sequence in one or more languages using a pretrained neural multilingual sequence-to-sequence (seq2seq) language model, the seq2seq model being pretrained for translating from a sequence in a source language to a sequence in a target language, wherein the source and target languages are taken from a set of languages for which the seq2seq model is pretrained, the seq2seq model comprising one or more trained denoising adapters for each of one or more languages in the set of languages, wherein the seq2seq model is fine-tuned using an unsupervised fine-tuning method comprising:

inserting, in a memory accessed by one or more processors, one or more denoising adapters for a new language into an encoder and/or a decoder of the pretrained seq2seq model, the new language not being in the set of languages; and

training, by one or more processors, parameters of each of the one or more denoising adapters for the new language on a language-specific denoising task using monolingual text for the new language.

27. The method of claim 26 , further comprising:

fine-tuning, by one or more processors, cross-attention weights of the seq2seq model on a translation task in one or more of the languages in the set of languages with parallel data while the trained denoising adapters for each of the one or more languages in the set of languages and the trained denoising adapters for the new language are inserted in the encoder and/or decoder of the seq2seq model;

wherein fine-tuning does not use parallel data for the new language.

28. The method of claim 26 , wherein said training parameters of each of the one or more denoising adapters is performed while freezing all parameters of the seq2seq model other than the parameters of the one or more denoising adapters and weights of an output projection layer in the seq2seq model.

29. The method of claim 26 , wherein the seq2seq model comprises a transformer encoder and an autoregressive transformer decoder, wherein the transformer decoder comprises a cross-attention layer;

wherein the method further comprises:

fine-tuning, by one or more processors, cross-attention weights of the seq2seq model on a translation task in one or more of the languages in the set of languages with parallel data while the trained denoising adapters for each of the one or more languages in the set of languages and the trained denoising adapters for the new language are inserted in the encoder and/or decoder of the seq2seq model;

wherein said fine-tuning comprises fine-tuning cross-attention weights of the cross-attention layer.

30. The method of claim 26 , wherein the denoising adapters learn language-specific encoding and decoding in the new language by denoising the monolingual texts in the new language.

31. The method of claim 30 , wherein the denoising comprises performing a noise function, the noise function comprising one more of: token masking, token deletion, word-span masking, sentence permutation, or document rotation.

32. The method of claim 26 , further comprising:

pretraining, by one or more processors, the seq2seq model, wherein said pretraining comprises:

inserting one or more denoising adapters for each of one or more languages in the one or more languages in the set of languages into an encoder and/or a decoder of the seq2seq model;

training, by one or more processors, parameters of each of the inserted one or more denoising adapters on a language-specific denoising task using monolingual text for each of the one or more languages in the set of languages; and

fine-tuning, by one or more processors, cross-attention weights of the seq2seq model on a translation task in at least one of the one or more languages in the set of languages with parallel data while the trained denoising adapters for all of the one or more languages in the set of languages are inserted in the encoder and/or decoder of the seq2seq model.

33. A method for translating an input sequence from a first language to a second language, comprising:

receiving a sequence in the first language and a neural multilingual sequence-to-sequence (seq2seq) model; and

converting the sequence in the first language to a sequence in the second language using the neural multilingual sequence-to-sequence (seq2seq) model;

wherein the neural multilingual sequence-to-sequence (seq2seq) model is trained by:

inserting, in a memory accessed by one or more processors, one or more denoising adapters for each of the first and the second languages into an encoder and/or a decoder of the seq2seq model;

training, by one or more processors, parameters of each of the one or more denoising adapters on a language-specific denoising task using monolingual text for each of the first and the second languages; and

fine-tuning, by one or more processors, cross-attention weights of the seq2seq model with the denoising adapter layers and the trained parameters on a translation task in at least one of the first and the second languages with parallel data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 30, 2023
From: BÉRARD, ALEXANDRE; BESACIER, LAURENT; GALLÉ, MATTHIAS; ÜSTÜN, AHMET
To: NAVER CORPORATION
Reel/Frame 064756/0593 →
Continuity (2)
Provisional Application 63266243 · Dec 30, 2021
Related Publication 20230214605A1 · Jul 6, 2023
References Cited (42)
US 20220147721A1 · Galle · 2022 [cited by examiner]
Guo, Junliang, et al. “Incorporating bert into parallel sequence decoding with adapters.” Advances in Neural Information Processing Systems 33 (2020): 10843-10854. (Year: 2020). [cited by examiner]
Aharoni, R., et al., “Massively Multilingual Neural Machine Translation,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Jun. 2-7, 2019, pp. 3874-3884. [cited by applicant]
Arivazhagan, N., et al., “Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges,” published on arXiv.org as 1907.05019, Jul. 11, 2019, 27 pages. [cited by applicant]
Artetxe, M., et al., “An Effective Approach to Unsupervised Machine Translation,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy: Jul. 28-Aug. 2, 2019, pp. 194-2… [cited by applicant]
Artetxe, M., et al., “Unsupervised Neural Machine Translation,” published on arXiv.org as 1770.11041, Feb. 26, 2018, 12 pages. [cited by applicant]
Attardi, G., “Wikiextractor,” retrieved on the Internet at: https://github.com/attardi/wikiextractor., 2015, 1 page. [cited by applicant]
Bapna, A., et al., “Simple, Scalable Adaptation for Neural Machine Translation,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur… [cited by applicant]
Conneau, A., et al., “Unsupervised Cross-Lingual Representation Learning at Scale,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul. 5-10, 2020 pp. 8440-8451. [cited by applicant]
Firat, O., et al., “Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism,” Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistic… [cited by applicant]
Garcia, X., et al., “Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages,” published on arXiv.org as 2009.11201, Mar. 12, 2021, 12 pages. [cited by applicant]
Guzmán, F., et al., “The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and t… [cited by applicant]
Ha, T., et al., “Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder,” published on arXiv.org as 1611.04798, Nov. 15, 2016, 10 pages. [cited by applicant]
Houlsby, N., et al., “Parameter-Efficient Transfer Learning for NLP,” proceedings of the 36th International Conference on Machine Learning 2019, published on arXiv.org as 1902.00751, Jun. 13, 2019, 13 pages. [cited by applicant]
Johnson, M., et al., “Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation,” Transactions of the Association for Computational Linguistics 5, Dec. 2017, pp. 339-351. [cited by applicant]
Kingma, D., et al., “Adam: A Method for Stochastic Optimization,” published as a conference paper at ICLR 2015, available on arXiv.org as 1412.6980, Jan. 30, 2017, 15 pages. [cited by applicant]
Kudo, T., et al., “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing,” Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: … [cited by applicant]
Kunchukuttan, A., et al., “The IIT Bombay English-Hindi Parallel Corpus,” published on arXiv.org as 1710.02855, May 19, 2018, 4 pages. [cited by applicant]
Lample, G., et al., “Cross-Lingual Language Model Pretraining,” published on arXiv.org as 1901.07291, Jan. 22, 2019, 10 pages. [cited by applicant]
Lample, G., et al., “Unsupervised Machine Translation Using Monolingual Corpora Only,” published as a conference paper at ICRL 2018, available on arXiv.org as 1711.00043, Apr. 13, 2018, 14 pages. [cited by applicant]
Lewis, M., et al., “BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguis… [cited by applicant]
Li, Z., et al., “Reference Language Based Unsupervised Neural Machine Translation,” Findings of the Association for Computational Linguistics: EMNLP, Nov. 16-20, 2020, pp. 4151-4162. [cited by applicant]
Liu, Y., et al., “Multilingual Denoising Pre-Training for Neural Machine Translation,” Transactions of the Association for Computational Linguistics 8, Nov. 2020, pp. 726-742. [cited by applicant]
Ott, M., et al., “Fairseq: A Fast, Extensible Toolkit for Sequence Modeling,” Proceedings of the 2019 Conference of the North, Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2-7, 2019, pp. 48-53. [cited by applicant]
Ott, M., et al., “Scaling Neural Machine Translation,” Proceedings of the Third Conference on Machine Translation: Research Papers, Oct. 31-Nov. 1, 2018, pp. 1-9. [cited by applicant]
Pfeiffer, J., et al., “MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer,” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online: Association for Co… [cited by applicant]
Philip, J., et al., “Monolingual Adapters for Zero-Shot Neural Machine Translation,” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 16-20, 2020, pp. 4465-4470. [cited by applicant]
Popović, M. “CHRF: Character n-gram F-Score for automatic MT evaluation,” Proceedings of the Tenth Workshop on Statistical Machine Translation, Sept. 17-18, 2015, pp. 392-395. [cited by applicant]
Post, M., “A Call for Clarity in Reporting BLEU Scores,” Proceedings of the Third Conference on Machine Translation: Research Papers, Oct. 31-Nov. 1, 2018, pp. 186-191. [cited by applicant]
Qi, Y., et al., “When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?,” Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: H… [cited by applicant]
Rebuffi, S., et al., “Efficient Parametrization of Multi-Domain Deep Neural Networks,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 8119-8127. [cited by applicant]
Rei, R., et al., “COMET: A Neural Framework for MT Evaluation,” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 16-20, 2020, pp. 2685-2702. [cited by applicant]
Sen, S., et al., “Multilingual Unsupervised NMT Using Shared Encoder and Language-Specific Decoders,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jul. 28-Aug. 2, 2019, pp. 30… [cited by applicant]
Sennrich, R., et al., “Improving Neural Machine Translation Models with Monolingual Data,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), Aug. 7-12, 2016, … [cited by applicant]
Stickland, A., et al., “BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning,” Proceedings of the 36th International Conference on Machine Learning, 2019, 10 pages. [cited by applicant]
Stickland, A., et al., “Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation,” Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lingu… [cited by applicant]
Sun, H., et al., “Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul. 5-10, 2020, pp. 3525-3535. [cited by applicant]
Tang, Y., et al., “Multilingual Translation with Extensible Multilingual Pretraining and Finetuning,” published on arXiv.org as 2008.00401, Aug. 2, 2020, 15 pages. [cited by applicant]
Üstün, A., et al., “UDapter: Language Adaptation for Truly Universal Dependency Parsing,” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 16-20, 2020, pp. 2302-2315. [cited by applicant]
Vaswani, A., et al., “Attention Is All You Need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), available on arXiv.org as 1706.03762, Dec. 6, 2017, 15 pages. [cited by applicant]
Wang, M., et al., “Cross-Lingual Supervision Improves Unsupervised Neural Machine Translation,” published on arXiv:2004.03137v3, Apr. 1, 2021, 8 pages. [cited by applicant]
Zhang, T., et al., “Bertscore: Evaluating text generation with BERT,” International Conference on Learning Representations, 2020, 43 pages. [cited by applicant]