IP Library › Granted Patent US 12,293,284
Granted Patent B2
US 12,293,284 · App. 17/136,054 · Granted May 6, 2025

Meta cooperative training paradigms

Inventors: Dingcheng Li (Sammamish, WA); Haiyan Yin (Singapore, SG); Xu Li (Beijing, CN); Ping Li (Bellevue, WA)
Assignee: Baidu USA, LLC
G06N3/08G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,293,284
App. No.
17/136,054
Granted
May 6, 2025
Kind
B2
Abstract

Generative adversarial models have several benefits; however, due to mode collapse, these generators face a quality-diversity trade-off (i.e., the generator models sacrifice generation diversity for increased generation quality). Presented herein are embodiments that improve the performance of adversarial content generation by decelerating mode collapse. In one or more embodiments, a cooperative training paradigm is employed where a second model is cooperatively trained with the generator and helps efficiently shape the data distribution of the generator against mode collapse. Moreover, embodiments of a meta learning mechanism may be used, where the cooperative update to the generator serves as a high-level meta task and which helps ensures the generator parameters after the adversarial update stay resistant against mode collapse. In experiments, tested employments demonstrated efficient slowdown of mode collapse for the adversarial text generators. Overall, embodiments outperformed the baseline approaches with significant margins in terms of both generation quality and diversity.

Claims (96)

1. A computer-implemented method for training a generator comprising:

responsive to a stop condition having not been reached, performing steps comprising:

sampling a set of data from a training data;

using a generator model, which comprises a set of generator parameter values, to generate a set of generated data;

computing an adversarial loss for the generator model using an adversarial training loss function;

determining a set of intermediate generator parameter values for the generator model using the adversarial loss and gradient descent;

using a set of data sampled from the training data as inputs:

into a second neural network model, which comprises a second neural network model set of parameter values, to obtain one or more output distributions from the second neural network model; and

into the generator model comprising the set of intermediate generator parameter values to obtain one or more output distributions from the generator model;

determining a meta gradient for a cooperate training loss that comprises comparing one or more output distributions from the second neural network model with one or more corresponding output distributions from the generator model;

updating a set of generator parameter values using an adversarial gradient, which is obtained using the adversarial loss for the generator model, and the meta gradient;

updating a set of discriminator parameter values for a discriminator model using an adversarial loss for the discriminator model; and

updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss for the second neural network model; and

responsive to the stop condition having been reached, outputting the generator model, which comprises a final updated set of generator parameter values.

2. The computer-implemented method of claim 1 further comprising as initial steps:

initializing at least the set of generator parameter values of the generator model and the set of discriminator parameter values of the discriminator model; and

pretraining the generator model using training data, the generator model, and the discriminator model.

3. The computer-implemented method of claim 2 wherein the second neural network model and the generator model share a same neural network structure and the method further comprises:

using at least some of the set of generator parameter values from the pretrained generator model as parameter values for the second neural network model.

4. The computer-implemented method of claim 1 wherein the step of updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss comprises:

using a maximum likelihood estimation (MLE) loss function.

5. The computer-implemented method of claim 4 wherein the step of updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss comprises:

minimizing Kullback-Leibler divergence between:

one or more outputs from the second neural network model using a set of data sampled from the training data; and

one or more outputs from the second neural network model using a mixture of data sampled from the training data and data sampled from data that were generated by the generator model.

6. The computer-implemented method of claim 5 wherein the mixture comprises an equal number or approximately equal number of data from the training data and data points which were generated by the generator model.

7. The computer-implemented method of claim 1 wherein the adversarial loss for the discriminator model and the adversarial loss for the generator model are obtained by using a min-max loss function.

8. A system comprising:

one or more processors; and

a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

responsive to a stop condition having not been reached, performing steps comprising:

sampling a set of data from a training data having a first distribution;

using a generator model, which comprises a set of generator parameter values, to generate a set of generated data;

computing an adversarial loss for the generator model using an adversarial training loss function;

determining a set of intermediate generator parameter values for the generator model using the adversarial loss and gradient descent;

using a set of data sampled from the training data as inputs:

into a second neural network model, which comprises a second neural network model set of parameter values, to obtain one or more output distributions from the second neural network model; and

into the generator model comprising the set of intermediate generator parameter values to obtain one or more output distributions from the generator model;

determining a meta gradient for a cooperate training loss that comprises comparing one or more output distributions from the second neural network model with one or more corresponding output distributions from the generator model;

updating a set of generator parameter values using an adversarial gradient, which is obtained using the adversarial loss for the generator model, and the meta gradient;

updating a set of discriminator parameter values for a discriminator model using an adversarial loss for the discriminator model; and

updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss for the second neural network model; and

responsive to the stop condition having been reached, outputting the generator model, which comprises a final updated set of generator parameter values.

9. The system of claim 8 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

initializing at least the set of generator parameter values of the generator model and the set of discriminator parameter values of the discriminator model; and

pretraining the generator model using training data, the generator model, and the discriminator model.

10. The system of claim 9 wherein the second neural network model and the generator model share a same neural network structure and the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

using at least some of the set of generator parameter values from the pretrained generator model as parameter values for the second neural network model.

11. The system of claim 8 wherein the step of updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss comprises:

using a maximum likelihood estimation (MLE) loss function.

12. The system of claim 11 wherein the step of updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss comprises:

minimizing Kullback-Leibler divergence between:

one or more outputs from the second neural network model using a set of data sampled from the training data; and

one or more outputs from the second neural network model using a mixture of data sampled from the training data and data sampled from data that were generated by the generator model.

13. The system of claim 12 wherein the mixture comprises an equal number or approximately equal number of data from the training data and data which were generated by the generator model.

14. The system of claim 8 wherein the adversarial loss for the discriminator model and the adversarial loss for the generator model are obtained by using a min-max loss function.

15. A computer-implemented method for training a generator comprising:

responsive to a stop condition having not been reached, performing steps comprising:

generating a set of generated data using a set of data from a training dataset comprising real data and a generator model from a generative adversarial system comprising the generator model having a set of generator model parameter values and a discriminator model having a set of discriminator parameter values;

computing an adversarial loss for the generator model using an adversarial training loss function;

determining a set of intermediate generator parameter values for the generator model using the adversarial loss and gradient descent;

using the generator model with the set of intermediate generator parameter values and a second neural network model to cooperatively train the generator model to decelerate mode collapse of the generator model, in which cooperative training comprises relating one or more output distributions of the generator model with one or more corresponding output distributions of the second neural network model;

updating a set of discriminator parameter values for the discriminator model using an adversarial loss for the discriminator model; and

updating a set of parameter values for the second neural network model using a cooperative training loss for the second neural network model; and

responsive to the stop condition having been reached, outputting the generator model;

wherein the step of using the generator model with the set of intermediate generator parameter values and a second neural network model to cooperatively train the generator model to decelerate mode collapse of the generator model comprises:

computing a cooperative training loss for the generator model using the set of data sampled from the training dataset comprising real data as inputs into the second neural network model and into the generator model comprising the set of intermediate generator parameter values;

determining a meta gradient using the cooperate training loss for the generator model; and

updating the set of generator parameter values using an adversarial gradient, which is obtained using the adversarial loss for the generator model, and the meta gradient.

16. The computer-implemented method of claim 15 further comprising as initial steps:

initializing at least the set of generator parameter values of the generator model and the set of discriminator parameter values of the discriminator model;

pretraining the generator model using training data and the generator model and the discriminator model; and

using at least some of the set of generator parameter values from the pretrained generator model as parameter values for the second neural network model.

17. The computer-implemented method of claim 15 wherein the generator is a natural language text generator and the second neural network model is a language model.

18. The computer-implemented method of claim 15 wherein the step of updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss comprises:

minimizing Kullback-Leibler divergence between:

one or more output distributions from the second neural network model using the set of data sampled from the training dataset comprising real data, and

one or more output distributions from the second neural network model using a mixture of data sampled from the training dataset comprising real data and data sampled from data that were generated by the generator model.

19. The computer-implemented method of claim 18 wherein the mixture comprises an equal number or approximately equal number of data from the training dataset and data which were generated by the generator model.

20. A system comprising:

one or more processors; and

a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:

responsive to a stop condition having not been reached, performing steps comprising:

generating a set of generated data using a set of data from a training dataset comprising real data and a generator model from a generative adversarial system comprising the generator model having a set of generator model parameter values and a discriminator model having a set of discriminator parameter values;

computing an adversarial loss for the generator model using an adversarial training loss function;

determining a set of intermediate generator parameter values for the generator model using the adversarial loss and gradient descent;

using the generator model with the set of intermediate generator parameter values and a second neural network model to cooperatively train the generator model to decelerate mode collapse of the generator model, in which cooperative training comprises relating one or more output distributions of the generator model with one or more corresponding output distributions of the second neural network model;

updating a set of discriminator parameter values for the discriminator model using an adversarial loss for the discriminator model; and

updating a set of parameter values for the second neural network model using a cooperative training loss for the second neural network model; and

responsive to the stop condition having been reached, outputting the generator model;

wherein the step of using the generator model with the set of intermediate generator parameter values and a second neural network model to cooperatively train the generator model to decelerate mode collapse of the generator model comprises:

computing a cooperative training loss for the generator model using the set of data sampled from the training dataset comprising real data as inputs into the second neural network model and into the generator model comprising the set of intermediate generator parameter values;

determining a meta gradient using the cooperate training loss for the generator model; and

updating the set of generator parameter values using an adversarial gradient, which is obtained using the adversarial loss for the generator model, and the meta gradient.

21. The system of claim 20 wherein the step of updating the second neural network model set of parameter values of the second neural network model using a cooperative training loss comprises:

minimizing Kullback-Leibler divergence between the second neural network model using the set of data sampled from the training dataset comprising real data and the second neural network model using a mixture of data sampled from the training dataset comprising real data and data sampled from data that were generated by the generator model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 4, 2021
From: LI, DINGCHENG; YIN, HAIYAN; LI, XU; LI, PING
To: BAIDU USA LLC
Reel/Frame 054803/0742 →
Continuity (2)
Provisional Application 62970638 · Feb 5, 2020
Related Publication 20210241099A1 · Aug 5, 2021
References Cited (47)
US 10140544B1 · Zhao · 2018 [cited by examiner]
US 20190122072A1 · Cricrì · 2019 [cited by examiner]
US 20200097554A1 · Rezagholizadeh · 2020 [cited by examiner]
US 20200336507A1 · Lee · 2020 [cited by examiner]
US 20210073630A1 · Zhang · 2021 [cited by examiner]
US 20210303925A1 · Hofmann · 2021 [cited by examiner]
Ngoc-Trung Tran et.al. (hereinafter Tran) Dist-GAN: An Improved GAN using Distance Constraints, Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 370-385., (Year: 2018). [cited by examiner]
Sutton et al., “Policy gradient methods for reinforcement learning with function approximation,” In NIPS, 1057-1063, 2000. (7pgs). [cited by applicant]
Ulyanov et al., “Instance Normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022, 2017. (6pgs). [cited by applicant]
Wen et al., “Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems,” arXiv preprint arXiv:1508.01745, 2015. (11pgs). [cited by applicant]
R. J. Williams “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning 8(3-4):229-256, 1992 (28 pgs). [cited by applicant]
Xu et al., “DP-GAN: diversity-promoting generative adversarial network for generating informative and diversified text,” arXiv preprint arXiv:1802.01345v2, 2018. (8pgs). [cited by applicant]
Yin et al., “Knowledge transfer for deep reinforcement learning with hierarchical experience replay,” In AAAI, 1640-1646, 2017. (7pgs). [cited by applicant]
Yu et al.,“SeqGAN: Sequence generative adversarial nets with policy gradient,” In AAAI, 2852-2858, 2017. (7pgs). [cited by applicant]
Zhang et al., “Adversarial feature matching for text generation,” In ICML, 2017. (10pgs). [cited by applicant]
Zhu et al., “Texygen: A benchmarking platform for text generation models,” arXiv preprint arXiv:1802.01886, 2018. (4pgs). [cited by applicant]
Xu et al., “Diversity-Promoting Gan: A Cross-Entropy Based Generative Adversarial Network for Diversified Text Generation,” arXiv preprint arXiv:1802.01345v3, 2018. (10pgs). [cited by applicant]
Guo et al., “Long text generation via adversarial training with leaked information,” In AAAI, 2018. (8 pgs). [cited by applicant]
Hinton et al., “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. (9 pgs). [cited by applicant]
Jang et al., “Categorical reparameterization with gumbel-softmax,” In ICLR, 2017. (12 pgs). [cited by applicant]
Kingma et al., Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2017. (15pgs). [cited by applicant]
Kusner et al., “Gans for sequences of discrete elements with the gumbel-softmax distribution,” arXiv preprint arXiv:1611.04051, 2016. (6 pgs). [cited by applicant]
Lamb et al., “Professor forcing: A new algorithm for training recurrent networks,” In NIPS, 2016. (9 pgs). [cited by applicant]
Li et al.,“Deep reinforcement learning for dialogue generation,” In EMNLP, 2016. (11 pgs). [cited by applicant]
Li et al., “Deep recurrent generative decoder for abstractive text summarization,” In EMNLP, 2091-2100, 2017. (10 pgs). [cited by applicant]
Li et al., “Learning to generalize: Meta-learning for domain generalization,” In AAAI, 3490-3497, 2018. (8 pgs). [cited by applicant]
Li et al., “Multi-agent discussion mechanism for natural language generation,” In AAAI, 6096-6103, 2019. (8 pgs). [cited by applicant]
Al-Shedivat et al., “Continuous adaptation via meta-learning in nonstationary and competitive environments,” In ICLR, 2018. (21pgs). [cited by applicant]
Arjovsky et al., “Wasserstein generative adversarial networks,” In ICML, 2017. (10 pgs). [cited by applicant]
Bahdanau et al., “Neural machine translation by jointly learning to align and translate,” In ICLR, 2015. (15pgs). [cited by applicant]
Caccia et al., “Language gans falling short,” arXiv preprint arXiv:1811.02549, Feb. 19, 2020. (18pgs). [cited by applicant]
Che et al., “Maximum-likelihood augmented discrete generative adversarial networks,” arXiv preprint arXiv:1702.07983, 2017. (11pgs). [cited by applicant]
Che et al., “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325, 2015. (7pgs). [cited by applicant]
Che et al., “Adversarial text generation via feature-mover's distance,” In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018. (12pgs). [cited by applicant]
Fedus et al., “Maskgan: better text generation via filling in the ,” In ICLR, 2018. (17pgs). [cited by applicant]
Finn et al., “Model-agnostic meta-learning for fast adaptation of deep networks,” In ICML, 2017. (10 pgs). [cited by applicant]
Goodfellow et al., “Generative adversarial nets,” In NIPS, 2014. (9 pgs). [cited by applicant]
Lin et al., “Adversarial ranking for language generation,” In NIPS, 2017. (11 pgs). [cited by applicant]
Liu et al., “Query-Oriented multi-document summarization via unsupervised deep learning,” Proceedings of the 26th AAAI Conference on Artificial Intelligence, 2012. (7pgs). [cited by applicant]
Lu et al., “Neural Text Generation: Past, Present and Beyond,” arXiv preprint arXiv:1803.07133, 2018. (7pgs). [cited by applicant]
Lu et al., “CoT: Cooperative Training for Generative Modeling of Discrete Data,”Proceedings of the 36 th International Conference on Machine Learning, 2019. (9 pgs). [cited by applicant]
Nie et al., “Relgan: Relational generative adversarial networks for text generation,” In ICLR, 2019. (20pgs). [cited by applicant]
Radford et al., “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434, 2016. (16pgs). [cited by applicant]
Rajeswar et al., “Adversarial generation of natural language,” arXiv preprint arXiv:1705.10929, 2017. (11pgs). [cited by applicant]
Rusu et al.,“Policy Distillation,” arXiv preprint arXiv:1511.06295, 2016. (13pgs). [cited by applicant]
Sun et al., “Logician and orator: Learning from the duality between language and knowledge in open domain,” In EMNLP, 2119-2130, 2018. (12pgs). [cited by applicant]
Sutskever et al., “Sequence to Sequence Learning with Neural Networks,” In NIPS, 2014. (9pgs). [cited by applicant]