IP Library › Granted Patent US 12,524,630
Granted Patent B2
US 12,524,630 · App. 17/463,084 · Granted Jan 13, 2026

Adversarial generation method for training a neural model

Inventors: Stéphane Clinchant (Meylan, FR); Badr Youbi Idrissi (Gagnac sur Garonne, FR)
Assignee: NAVER CORPORATION
G06F40/58
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,524,630
App. No.
17/463,084
Granted
Jan 13, 2026
Kind
B2
Abstract

Methods and systems for training a neural language model. Clean sequence pairs are received including clean source and target sequences. For each clean sequence pair, a noisy version is sampled with an adversarial generator to generate a noisy sequence pair. Parameters of the neural language model are optimized on the clean and noisy sequence pairs. Parameters of the adversarial generator are optimized to minimize a modeling loss of the adversarial generator and maximize a neural language loss of the neural language model using backpropagation.

Claims (113)

1 . A method for training a neural language model, the method being implemented by a processor and memory, the method comprising:

receiving a plurality of clean sequence pairs, each clean sequence pair including a clean source sequence and a clean target sequence, the plurality of clean sequence pairs providing a clean parallel corpus; and

training an adversarial generator to generate a noisified parallel corpus from the clean parallel corpus while training the neural language model using both the noisified parallel corpus and the clean parallel corpus to improve robustness of the neural language model;

wherein said training the adversarial generator comprises:

(i) for each clean sequence pair, generating a noisy version with the adversarial generator to generate a noisy sequence pair, the adversarial generator being a differentiable model definable by a plurality of parameters;

(ii) determining a modeling loss of the adversarial generator using at least the generated noisy sequence pairs; and

(iii) optimizing the parameters of the adversarial generator based on at least the determined modeling loss;

wherein said training the neural language model using both the noisified parallel corpus and the clean parallel corpus comprises:

(i) inputting the clean sequence pairs and the noisy sequence pairs to the neural language model, wherein the neural language model generates output sequences from the clean sequence pairs and from the noisy sequence pairs;

(ii) determining a neural language model loss using the generated output sequences from the clean sequence pairs and from the noisy sequence pairs; and

(iii) optimizing parameters of the neural language model based on the determined neural language loss.

2 . The method of claim 1 , wherein the neural language model comprises a machine translation model; and

wherein said optimizing the parameters of the adversarial generator is based on objectives comprising minimizing the modeling loss of the adversarial generator and maximizing a neural language model loss of the neural language model using backpropagation.

3 . The method of claim 2 , wherein the machine translation model is bilingual.

4 . The method of claim 2 , wherein the clean sequence pairs comprise clean sentence pairs, the noisy sequence pairs comprise noisy sentence pairs, the plurality of clean sentence pairs is received from a machine translation dataset, the clean source sequence comprises a clean source sentence in a first language, and the clean target sequence comprises a clean target sentence in a second language.

5 . The method of claim 4 , wherein the machine translation model comprises an encoder and a decoder.

6 . The method of claim 5 , wherein the machine translation model comprises a transformer model.

7 . The method of claim 5 ,

wherein training the neural language model improves robustness of the machine translation model to token substitutions,

wherein the adversarial generator is trained to generate token substitutions for tokens in the clean sentence pair that are linguistically plausible according to a language model objective, and that are detrimental to the machine translation model according to an adversarial objective, and

wherein the tokens comprise words and/or subwords.

8 . The method of claim 7 , wherein the adversarial generator is further trained based on an offset weighting the language model objective and/or the adversarial objective.

9 . The method of claim 5 ,

wherein said generating a noisy version comprises sampling;

wherein the method further comprises:

concatenating each clean sequence pair before said sampling.

10 . The method of claim 9 , wherein the adversarial generator further comprises a differentiable model using a continuous probability distribution that approximates categorical samples.

11 . The method of claim 9 , wherein the differential model comprises a Gumbel-softmax reparametrization model.

12 . The method of claim 5 , further comprising:

inputting a source sentence from the noisy sentence pair to the encoder and inputting a target sentence from the noisy sentence pair to the decoder;

the encoder outputting a dense vector representation; and

the decoder receiving the dense vector representation.

13 . The method of claim 1 , further comprising:

further optimizing parameters of the neural language model using a discriminator objective of a discriminator configured to distinguish between clean and substituted tokens in the noisy sequence pairs.

14 . The method of claim 13 ,

wherein the neural language model comprises an encoder and a decoder; and

wherein the discriminator is provided at an end of the encoder and/or at an end of the decoder.

15 . The method of claim 9 , further comprising preprocessing the plurality of clean sequence pairs, wherein the preprocessing comprises:

splitting the clean sequence pairs into tokens, the tokens comprising words and/or subwords;

randomly masking one or more of the tokens in the clean source sentence and/or the clean target sentence;

wherein said sampling a noisy version of the clean sentence pair perturbates the masked tokens.

16 . The method of claim 15 , wherein said sampling a noisy version replaces the masked tokens with an objective of preserving meaning of the masked token.

17 . The method of claim 16 , wherein said preprocessing and said sampling take place during embedding of tokens in the clean sequence pairs.

18 . A method for generating an output sequence from an input sequence, the method comprising:

receiving the input sequence;

processing the input sequence with the neural language model trained using the method of claim 1 , wherein the trained neural language model generates the output sequence; and

transmitting the output sequence.

19 . The method of claim 18 , wherein the neural language model comprises a bilingual machine translation model, the bilingual machine translation model comprising:

an encoder for receiving the input sequence and generating a dense vector representation; and

a decoder for receiving the dense vector representation and generating the output sequence;

wherein the input sequence is in a first language and the output sequence is in a second language.

20 . A system for training a machine translation model, the system being implemented by a processor and a memory, the system comprising:

a machine translation model; and

a multilingual language model generator that is trained to generate token substitutions in clean sentence pairs of source and target sentences in a machine translation batch to provide noisy sentence pairs, the tokens comprising words and/or subwords, the token substitutions comprising insertions and deletions, the token substitutions being linguistically plausible based on a first objective and detrimental to the machine translation model based on a second objective, the multilingual language model generator being differentiable;

wherein the multilingual language model generator is trained to generate a noisified parallel corpus of a clean parallel corpus while the machine translation model is trained to improve robustness of the machine translation model to token substitutions by:

for each clean sentence pair in the clean parallel corpus, sampling a noisy version with the multilingual language model generator;

inputting the clean sentence pairs and their noisy version to the machine translation model;

optimizing machine translation parameters both on the clean sentence pairs and on their noisy version, wherein the machine translation model generates output sequences from the clean sentence pairs and from their noisy version, and said optimizing parameters is based on machine translation losses determined for the generated output sequences; and

optimizing parameters of the multilingual language model generator using backpropagation to minimize the language modeling loss based on the first objective and maximize the machine translation loss based on the second objective.

21 . The system of claim 20 , wherein the machine translation model comprises a transformer model including an encoder and a decoder.

22 . The system of claim 21 , further comprising:

a preprocessor configured to:

receive the clean sentence pairs;

split the clean sentence pairs into common tokens, the tokens comprising words and/or subwords, and

mask random tokens in the clean sentence pairs;

wherein the generated substitutions replace the masked random tokens.

23 . The system of claim 22 , wherein a discriminator is provided at an end of the encoder and at an end of the decoder.

24 . The system of claim 23 , wherein the noisy sentence pair comprises a noisy source sentence and/or a noisy target sentence;

wherein said optimizing machine translation parameters both on the clean sentence pairs and on their noisy version comprises:

inputting the noisy source sentence and/or the clean source sentence to the encoder, the encoder generating a vector representation and outputting the vector representation to the decoder;

inputting the noisy target sentence and/or the clean target sentence to the decoder, the decoder modeling a conditional probability of output tokens and generating the output sequence;

determining a translation loss from the generated output sequence of the decoder; and

optimizing the machine translation parameters based on the determined translation loss.

25 . The system of claim 20 , further comprising:

a discriminator for distinguishing between clean and substituted tokens in the noisy sequence pairs;

wherein the multilingual language model generator is further configured to further optimize machine translation parameters on a discriminator objective of the discriminator.

26 . A method for training a neural machine translation model, the method being implemented by a processor and memory, the method comprising:

noisifying, using an adversarial generator, at least one of a source side and a target side of a clean parallel corpus to provide a noisified parallel corpus, wherein the source side and the target side of the clean parallel corpus comprises clean sentence pairs including a clean source sentence and a clean target sentence, respectively, and wherein the adversarial generator generates noisy sentence pairs in the noisified parallel corpus including a noisy source sentence and a noisy target sentence;

training the neural machine translation model using both the noisified parallel corpus and the clean parallel corpus to improve robustness of the machine translation model to word substitutions; and

training the adversarial generator to generate the noisified parallel corpus;

wherein said training the machine translation model and said training the adversarial generator use backpropagation and take place in a same forward pass and backward pass;

wherein said training the neural machine translation model comprises:

inputting the clean sentence pairs and the noisy sentence pairs to the machine translation model; and

optimizing machine translation parameters both on the clean sentence pairs and on the noisy sentence pairs, wherein the machine translation model generates output sequences from the clean sentence pairs and from the noisy sentence pairs, and said optimizing parameters is based on machine translation losses determined for the generated output sequences.

27 . The method of claim 26 ,

wherein, in the generated noisy source sentence and the generated noisy target sentence, random subwords have been replaced by candidate words, the candidate words being both linguistically plausible and likely to be detrimental to the neural machine translation model.

28 . A method for training a neural machine translation model, the method being implemented by a processor and memory, the method comprising:

noisifying, using an adversarial generator, at least one of a source side and a target side of a clean parallel corpus to provide a noisified parallel corpus;

training the neural machine translation model using both the noisified parallel corpus and the clean corpus to improve robustness of the machine translation model to word substitutions; and

training the adversarial generator to generate the noisified parallel corpus;

wherein said training the machine translation model and said training the adversarial generator use backpropagation and take place in a same forward pass and backward pass;

wherein said neural machine translation model comprises an encoder and a decoder;

wherein said noisifying comprises the adversarial generator generating, during the forward pass, a noisy sentence pair in the noisified parallel corpus including a noisy source sentence and a noisy target sentence in which random subwords have been replaced by candidate words;

wherein the noisy source sentence is fed to the encoder and the noisy target sentence is fed to the decoder during the forward pass;

wherein the encoder generates a dense vector representation from the noisy target sentence and feeds the dense vector representation to the decoder during the forward pass; and

wherein the decoder generates an output sequence during the forward pass.

29 . The method of claim 28 , wherein said training the neural machine translation model comprises, during the backward pass:

optimizing parameters of the neural machine translation model on the clean sentence pair and on the noisy sentence pair using backpropagation; and

further optimizing parameters of the neural language model using a discriminator objective of a discriminator configured to distinguish between real and fake examples using backpropagation.

30 . The method of claim 29 , wherein said training the adversarial generation comprises, during the backward pass:

optimizing parameters of the adversarial generator to minimize a modeling loss of the adversarial generator and maximize the translation loss using backpropagation.

31 . An apparatus for training a neural language model comprising:

a non-transitory computer-readable medium having executable instructions stored thereon for causing a processor and a memory to:

receive a plurality of clean sequence pairs, each clean sequence pair including a clean source sequence and a clean target sequence, the plurality of clean sequence pairs providing a clean parallel corpus; and

training an adversarial generator to generate a noisified parallel corpus from the clean parallel corpus while training the neural language model using both the noisified parallel corpus and the clean parallel corpus to improve robustness of the neural language model;

wherein said training the adversarial generator comprises:

(i) for each clean sequence pair, generating a noisy version with the adversarial generator to generate a noisy sequence pair, the adversarial generator being a differentiable model definable by a plurality of parameters;

(ii) determining a modeling loss of the adversarial generator using at least the generated noisy sequence pairs; and

(iii) optimizing the parameters of the adversarial generator based on at least the determined modeling loss;

wherein said training the neural language model using both the noisified parallel corpus and the clean parallel corpus comprises:

(i) inputting the clean sequence pairs and the noisy sequence pairs to the neural language model, wherein the neural language model generates output sequences from the clean sequence pairs and from the noisy sequence pairs;

ii determining a neural language model loss using the generated output sequences from the clean sequence pairs and from the noisy sequence pairs; and

iii) optimizing parameters of the neural language model based on the determined neural language loss.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 2, 2023
From: CLINCHANT, STÉPHANE; IDRISSI, BADR YOUBI
To: NAVER CORPORATION
Reel/Frame 062568/0831 →
Continuity (1)
Related Publication 20230084333A1 · Mar 16, 2023
References Cited (50)
US 10452978B2 · Shazeer et al. · 2019 [cited by applicant]
US 20200097554A1 · Rezagholizadeh · 2020 [cited by examiner]
US 20200210772A1 · Bojar · 2020 [cited by examiner]
US 20210241099A1 · Li · 2021 [cited by examiner]
Alzantot, M., et al., “Generating Natural Language Adversarial Examples,” Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Association for Computational Linguist… [cited by applicant]
Bahdanau, D., et al., “Neural Machine Translation by Jointly Learning to Align and Translate,” 3rd International Conference on Learning Representations, ICLR 2015—Conference Track Proceedings and 2015, published on arXi… [cited by applicant]
Belinkov, Y., et al., “Synthetic and Natural Noise both Break Neural Machine Translation,” International Conference on Learning Representations 2018, published on Arxiv.org as 1711.02173, Feb. 24, 2018, 13 pages. [cited by applicant]
Carlni, N., et al., “On Evaluating Adversarial Robustness,” published in Arxiv.org as 1902.06705v2, Feb. 20, 2019, 24 pages. [cited by applicant]
Cheng, M., et al., “Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 3601-3608. [cited by applicant]
Cheng, Y., et al., “Robust Neural Machine Translation with Doubly Adversarial Inputs,” ACL 2019—57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Florence Italy, Jul. 2… [cited by applicant]
Cheng, Y., et al., “AdvAug: Robust Adversarial Augmentation for Neural Machine Translation,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, Association for Computational… [cited by applicant]
Cheng, Y., et al., “Towards Robust Neural Machine Translation,” Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), Melbourne, Australia, Jul. 15-20, 2018, pp. … [cited by applicant]
Clark, K., et al., “Electra: Pre-Training Text Encoders as Discriminators Rather Than Generators,” International Conference on Learning Representations, 2020, 18 pages. [cited by applicant]
Clinchant, S., et al., “On the Use of Bert for Neural Machine Translation,” Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, Association for Computational Linguistics, Nov. 4, 2019, pp. 1… [cited by applicant]
Ebrahimi, J., et al., “On Adversarial Examples for Character-Level Neural Machine Translation,” Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, Association for C… [cited by applicant]
Ebrahimi, J., et al., “HotFlip: White-Box Adversarial Examples for Text Classification,” Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 2: Short Papers), Melbourne, Austral… [cited by applicant]
Goodfellow, I., et al., “Explaining and Harnessing Adversarial Examples,” 3rd International Conference on Learning Representations, ICLR 2015—Conference Track Proceedings. International Conference on Learning Representa… [cited by applicant]
Hochreiter, S., et al., “Long Short-Term Memory,” Neural Computation, 9(8), 1997, pp. 1735-1780. [cited by applicant]
Jang, E., et al., “Categorical Reparameterization with Gumbel-Softmax,” ICLR 2017, published on ArXiv.org as 1611.01144, Nov. 22, 2016, 12 pages. [cited by applicant]
Karpukhin, V., et al., “Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation,” Proceedings of 2019 EMNLP Workshop W-NUT: the 5th Workshop on Noisy User-Generated Text, Hong Kong, China… [cited by applicant]
Koehn, P., et al., “Six Challenges for Neural Machine Translation,” Proceedings of the First Workshop on Neural Machine Translation, Association for Computational Linguistics (ACL), Vancouver, Canada, Aug. 4, 2017, pp. … [cited by applicant]
Lample, G., et al., “Cross-Lingual Language Model Pretraining,” published on Arxiv.org as 1901.07291, Jan. 22, 2019. 10 pages. [cited by applicant]
Li, J., et al., “Understanding Neural Networks through Representation Erasure,” published on arXiv.org as 1612.08220, Jan. 10, 2017, 18 pages. [cited by applicant]
Maddison, C., et al., “The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables,” 5th International Conference on Learning Representations, ICLR 2017, published on Arxiv.org as 1611.00712, Mar. 5,… [cited by applicant]
Madry, A., et al., “Towards Deep Learning Models Resistant to Adversarial Attacks,” published on ArXiv.org as 1706.06083, Sep. 4, 2019, 23 pages. [cited by applicant]
Michel, P., et al., “On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human … [cited by applicant]
Miyato, T., et al., “Adversarial Training Methods for Semi-Supervised Text Classification,” published on ArXiv.org as 1605.077257 Nov. 7, 2016, 10 pages. [cited by applicant]
Miyato, T., et al., “Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence 41, No. 8, Aug. 2019, pp. 1979-1993. [cited by applicant]
Müller, M., et al., “Domain Robustness in Neural Machine Translation,” published on Arxiv.org as 1911.03109, Nov. 8, 2019, 11 pages. [cited by applicant]
Niu, X., et al., “Evaluating Robustness to Input Perturbations for Neural Machine Translation,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, Association for Computatio… [cited by applicant]
Ott, M., et al., FairSEQ: A Fast, Extensible Toolkit for Sequence Modeling Proceedings of the 2019 Conference of the North, Minneapolis, Minnesota, Association for Computational Linguistics, Jun. 2-7, 2019, pp. 48-53. [cited by applicant]
Papernot, N., et al., “Crafting Adversarial Input Sequences for Recurrent Neural Networks,” MILCOM 2016—2016 IEEE Military Communications Conference, Baltimore, MD, USA: IEEE, 2016, pp. 49-54. [cited by applicant]
Post, M., “A Call for Clarity in Reporting Bleu Scores,” Proceedings of the Third Conference on Machine Translation: Research Papers, Belgium, Brussels, Association for Computational Linguistics, Oct. 31-Nov. 1, 2018, p… [cited by applicant]
Ranzato, M., et al., “Sequence Level Training with Recurrent Neural Networks,” ICLR 2016, San Juan, Puerto Rico, Conference Track Proceedings, published on ArXiv.org as 1511.06732, May 6, 2016, 16 pages. [cited by applicant]
Rawlinson, G., “The Significance of Letter Position in Word Recognition,” IEEE Aerospace and Electronic Systems Magazine, 22(1), 2007, pp. 26-27. [cited by applicant]
Sato, M., et al., “Effective Adversarial Regularization for Neural Machine Translation,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, Association for Computat… [cited by applicant]
Shafahi, A., et al., “Adversarial Training for Free!,” 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 2019, 12 pages. [cited by applicant]
Shen, S., et al., “Minimum Risk Training for Neural Machine Translation,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), Berlin, Germany. Association for C… [cited by applicant]
Szegedy, C., et al., “Intriguing Properties of Neural Networks,” 2nd International Conference on Learning Representations, ICLR 2014, published on ArXiv.org as 1312.6199, Feb. 19, 2014, 10 pages. [cited by applicant]
Tramèr, F., et al., “Ensemble Adversarial Training: Attacks and Defenses,” International Conference on Learning Representations, 2018, 20 pages. [cited by applicant]
Tsipras, D., et al., “Robustness May Be at Odds With Accuracy,” International Conference on Learning Representations (ICLR), 2019, 23 pages. [cited by applicant]
Vaswani, A., et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems 30, NIPS 2017, Long Beach CA USA, 2017, pp. 5998-6008. [cited by applicant]
Wang, C., et al., “On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational … [cited by applicant]
Wong, E., et al., “Fast is Better Than Free: Revisiting Adversarial Training,” International Conference on Learning Representations (ICLR), 2020, 17 pages. [cited by applicant]
Wu, L., et al., “Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter,” Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgiu… [cited by applicant]
Yin, P., et al., “Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets,” Published on arXiv 1903.05662v4, on Sep. 25, 2019, 30 pages. [cited by applicant]
Zhang, D., et al., “You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle,” 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 2019, 12 pages. [cited by applicant]
Zhang, H., et al., “Theoretically Principled Trade-off between Robustness and Accuracy,” Proceedings of the 36th International Conference on Machine Learning, PMLR 97, 2019, pp. 7472-7482. [cited by applicant]
Zou, W., et al., “A Reinforced Generation of Adversarial Samples for Neural Machine Translation,” published on Arxiv.org as 1911.03677, Nov. 9, 2019, 8 pages. [cited by applicant]
Devlin, J., et al., Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu… [cited by applicant]
Cited By (1)
US 12,705,368