IP Library › Granted Patent US 12,307,377
Granted Patent B2
US 12,307,377 · App. 17/110,629 · Granted May 20, 2025

Generating data based on pre-trained models using generative adversarial models

Inventors: Wang Zhou (White Plains, NY); Yang Zhang (Cambridge, MA); Shiyu Chang (Elmsford, NY)
Assignee: International Business Machines Corporation
G06N3/088G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,377
App. No.
17/110,629
Granted
May 20, 2025
Kind
B2
Abstract

Techniques for generator model training are provided. A classifier model that was trained using one or more data samples in a target class is received, and a generative adversarial network (GAN) is trained to generate simulated data samples for the target class, comprising: generating a first simulated data sample using a generator model, computing a first discriminator loss by processing the first simulated data sample using a discriminator model, computing a classifier loss by processing the first simulated data sample using the classifier model, and refining the generator model based on the first discriminator loss and the classifier loss.

Claims (76)

1. A method, comprising:

receiving a classifier model that was trained using one or more data samples in belonging to a plurality of classes;

determining a target class of the plurality of classes; and

training a generative adversarial network (GAN) to generate simulated data samples belonging to the target class, comprising:

generating a random input vector;

generating a first simulated data sample based on processing only the random input vector using a generator model;

generating a first discriminator output by processing the first simulated data using a discriminator model, wherein the first discriminator output predicts whether the first simulated data sample was generated by the generator model;

computing a first discriminator loss based on whether the first discriminator output accurately classifies the first simulated data sample as having been generated by the generator model;

generating a second discriminator output by processing a first data samples from one or more non-target classes of the plurality of classes using the discriminator model, wherein the second discriminator output predicts whether the first data sample was generated by the generator model;

computing a second discriminator loss based on whether the second discriminator output accurately classifies the first data sample as not generated by the generator model;

generating a classifier output by processing the first simulated data sample using the classifier model, wherein the classifier output predicts whether the first simulated data sample belongs to the target class;

computing a classifier loss based on comparing the classifier output and the target class; and

refining the generator model based on the first discriminator loss, the second discriminator loss, and the classifier loss.

2. The method of claim 1 , wherein the GAN is trained without processing any data samples in the target class.

3. The method of claim 1 , wherein training the GAN further comprises:

generating a second simulated data sample using the generator model;

computing a third discriminator loss by processing the second simulated data sample using the discriminator model; and

refining the discriminator model based on the third discriminator loss.

4. The method of claim 1 , wherein the discriminator model is trained to differentiate between the simulated data samples generated by the generator model and data samples not generated by the generator model.

5. The method of claim 1 , the method further comprising:

generating an evaluation data sample for the target class by providing the generator model with a randomized input vector.

6. The method of claim 5 , the method further comprising:

determining that the classifier model was trained with insufficient data samples in the target class based at least in part on the evaluation data sample, wherein determining that the classifier model was trained with insufficient data comprises determining that accuracy of the classifier model with respect to the target class is below a threshold.

7. The method of claim 5 , the method further comprising:

determining that the data samples in the target class used to train the classifier model included one or more suspicious features based at least in part on the evaluation data sample.

8. One or more computer-readable storage media collectively containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising:

receiving a classifier model that was trained using one or more data samples in belonging to a plurality of classes;

determining a target class of the plurality of classes; and

training a generative adversarial network (GAN) to generate simulated data samples belonging to the target class, comprising:

generating a random input vector;

generating a first simulated data sample based on processing only the random input vector using a generator model;

generating a first discriminator output by processing the first simulated data using a discriminator model, wherein the first discriminator output predicts whether the first simulated data sample was generated by the generator model;

computing a first discriminator loss based on whether the first discriminator output accurately classifies the first simulated data sample as having been generated by the generator model;

generating a second discriminator output by processing a first data samples from one or more non-target classes of the plurality of classes using the discriminator model, wherein the second discriminator output predicts whether the first data sample was generated by the generator model;

computing a second discriminator loss based on whether the second discriminator output accurately classifies the first data sample as not generated by the generator model;

generating a classifier output by processing the first simulated data sample using the classifier model, wherein the classifier output predicts whether the first simulated data sample belongs to the target class;

computing a classifier loss based on comparing the classifier output and the target class; and

refining the generator model based on the first discriminator loss, the second discriminator loss, and the classifier loss.

9. The computer-readable storage media of claim 8 , wherein the GAN is trained without processing any data samples in the target class.

10. The computer-readable storage media of claim 8 , wherein training the GAN further comprises:

generating a second simulated data sample using the generator model;

computing a third discriminator loss by processing the second simulated data sample using the discriminator model; and

refining the discriminator model based on the third discriminator loss.

11. The computer-readable storage media of claim 8 , wherein the discriminator model is trained to differentiate between the simulated data samples generated by the generator model and data samples not generated by the generator model.

12. The computer-readable storage media of claim 8 , the operation further comprising:

generating an evaluation data sample for the target class by providing the generator model with a randomized input vector.

13. The computer-readable storage media of claim 12 , the operation further comprising:

determining that the classifier model was trained with insufficient data samples in the target class based at least in part on the evaluation data sample, wherein determining that the classifier model was trained with insufficient data comprises determining that accuracy of the classifier model with respect to the target class is below a threshold.

14. The computer-readable storage media of claim 12 , the operation further comprising:

determining that the data samples in the target class used to train the classifier model included one or more suspicious features based at least in part on the evaluation data sample.

15. A system comprising:

one or more computer processors; and

one or more memories collectively containing one or more programs which when executed by the one or more computer processors performs an operation, the operation comprising:

receiving a classifier model that was trained using one or more data samples belonging to a plurality of classes;

determining a target class of the plurality of classes; and

training a generative adversarial network (GAN) to generate simulated data samples belonging to the target class, comprising:

generating a random input vector;

generating a first simulated data sample based on processing only the random input vector using a generator model;

generating a first discriminator output by processing the first simulated data using a discriminator model, wherein the first discriminator output predicts whether the first simulated data sample was generated by the generator model;

computing a first discriminator loss based on whether the first discriminator output accurately classifies the first simulated data sample as having been generated by the generator model;

generating a second discriminator output by processing a first data samples from one or more non-target classes of the plurality of classes using the discriminator model, wherein the second discriminator output predicts whether the first data sample was generated by the generator model:

computing a second discriminator loss based on whether the second discriminator output accurately classifies the first data sample as not generated by the generator model;

generating a classifier output by processing the first simulated data sample using the classifier model, wherein the classifier output predicts whether the first simulated data sample belongs to the target class;

computing a classifier loss based on comparing the classifier output and the target class; and

refining the generator model based on the first discriminator loss, the second discriminator loss, and the classifier loss.

16. The system of claim 15 , wherein the GAN is trained without processing any data samples in the target class, and wherein training the GAN further comprises:

generating a second simulated data sample using the generator model;

computing a third discriminator loss by processing the second simulated data sample using the discriminator model; and

refining the discriminator model based on the third discriminator loss.

17. The system of claim 15 , wherein the discriminator model is trained to differentiate between the simulated data samples generated by the generator model and data samples not generated by the generator model.

18. The system of claim 15 , the operation further comprising:

generating an evaluation data sample for the target class by providing the generator model with a randomized input vector.

19. The system of claim 18 , the operation further comprising:

determining that the classifier model was trained with insufficient data samples in the target class based at least in part on the evaluation data sample, wherein determining that the classifier model was trained with insufficient data comprises determining that accuracy of the classifier model with respect to the target class is below a threshold.

20. The system of claim 18 , the operation further comprising:

determining that the data samples in the target class used to train the classifier model included one or more suspicious features based at least in part on the evaluation data sample.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 3, 2020
From: ZHOU, WANG; ZHANG, YANG; CHANG, SHIYU
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 054530/0345 →
Continuity (1)
Related Publication 20220180203A1 · Jun 9, 2022
References Cited (68)
US 9015093B1 · Commons · 2015 [cited by applicant]
US 9666184B2 · Lee et al. · 2017 [cited by applicant]
US 11087170B2 · Malaya · 2021 [cited by examiner]
US 11455495B2 · Nikolenko et al. · 2022 [cited by examiner]
US 20190279009A1 · Srirangam Narashiman et al. · 2019 [cited by applicant]
US 20210383274A1 · Takahashi et al. · 2021 [cited by applicant]
CN 110334806A · 2019 [cited by examiner]
CN 111476783A · 2020 [cited by applicant]
WO 2020084683A1 · 2021 [cited by applicant]
WO 2021208601A1 · 2021 [cited by applicant]
WO WO2019127231A1 · 2024 [cited by examiner]
Zilong Lin et al., IDSGAN: Generative Adversarial Networks for Attack Generation against Intrusion Detection, Jun. 2019, Shool of Cyber Security, Shangai Jiao Tong University. [cited by examiner]
Zilong Lin et al., IDSGAN: Generative Adversarial Networks for Attack Generation against Intrusion Detection, Jun. 2019, Shool of Cyber Security, Shangai Jiao Tong University (Year: 2019). [cited by examiner]
Kun Fang, “Classification Algorithm Optimization Based on Triple-GAN”, 2020 (Year: 2020). [cited by examiner]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. “Generative Adversarial Nets”. In Advances in Neural Information Processing Systems, pp. 2… [cited by applicant]
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. “Least Squares Generative Adversarial Networks”. In IEEE International Conference on Computer Vision, pp. 2794-2802, 2017. [cited by applicant]
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. “Improved techniques for training GANs”. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016. [cited by applicant]
Marc G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Remi Munos. “The Cramer Distance as a Solution to Biased Wasserstein Gradients”. arXiv preprint arXiv:1705.10743… [cited by applicant]
Martin Arjovsky, Soumith Chintala, and Leon Bottou. “Wasserstein GAN”. arXiv preprint arXiv:1701.07875, 2017. [cited by applicant]
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. “Improving GANs using Optimal transport”, In International Conference on Learning Representations, 2018. [cited by applicant]
Jae Hyun Lim and Jong Chul Ye. “Geometric GAN”. arXiv preprint arXiv:1705.02894, 2017. [cited by applicant]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. “f-GAN: Training Generative Neural Samplers Using Variational Divergence Minimization”. In Advances in Neural Information Processing Systems, pp. 271-279, 2016. [cited by applicant]
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. “On Convergence and Stability of GANs”. arXiv preprint arXiv:1705.07215, 2017. [cited by applicant]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017. [cited by applicant]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. “Spectral Normalization for Generative Adversarial Networks”. arXiv preprint arXiv:1802.05957, 2018. [cited by applicant]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. “Self-attention Generative Adversarial Networks”. arXiv preprint arXiv:1805.08318, 2018. [cited by applicant]
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. “Which Training Methods for GANs do Actually Converge”? arXiv preprint arXiv:1801.04406, 2018. [cited by applicant]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. “Progressive Growing of GANs for Improved Quality, Stability, and Variation”. arXiv preprint arXiv:1710.10196, 2017. [cited by applicant]
Tero Karras, Samuli Laine, and Timo Aila. “A style-based generator architecture for generative adversarial networks”. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019. [cited by applicant]
Mehdi Mirza and Simon Osindero. “Conditional generative adversarial nets”. arXiv preprint arXiv:1411.1784, 2014. [cited by applicant]
Augustus Odena, Christopher Olah, and Jonathon Shlens. “Conditional image synthesis with auxiliary classifier GANs”. In 34th International Conference on Machine Learning, pp. 2642-2651, 2017. [cited by applicant]
Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. “Twin auxilary classifiers GAN”. In Advances in Neural Information Processing Systems, pp. 1328-1337, 2019. [cited by applicant]
Takeru Miyato and Masanori Koyama. “cGANs with projection discriminator”. In International Conference on Learning Representations, 2018. [cited by applicant]
Andrew Brock, Jeff Donahue, and Karen Simonyan. “Large scale gan training for high fidelity natural image synthesis”. In International Conference on Learning Representations, 2019. [cited by applicant]
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. “Visualizing higher-layer features of a deep network”. University of Montreal Tech Report, 2009. [cited by applicant]
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. “Inceptionism: Going deeper into neural networks”. Google Research Blog, 2015. [cited by applicant]
Aravindh Mahendran and Andrea Vedaldi. “Visualizing deep convolutional neural networks using natural pre-images”. International Journal of Computer Vision, pp. 233-255, 2016. [cited by applicant]
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. “Deep inside convolutional networks: Visualising image classification models and saliency maps”. ICLR workshop, 2014. [cited by applicant]
Donglai Wei, Bolei Zhou, Antonio Torrabla, and William Freeman. “Understanding intra-class knowledge inside cnn” arXiv preprint arXiv:1507.02379, 2015. [cited by applicant]
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. “Feature visualization”. Distill, 2017. [cited by applicant]
Alexey Dosovitskiy and Thomas Brox. “Generating images with perceptual similarity metrics based on deep networks”. In Advances in Neural Information Processing Systems, pp. 658-666, 2016. [cited by applicant]
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks”. In Advances in Neural Information Processing S… [cited by applicant]
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. “Plug & play generative networks: Conditional iterative generation of images in latent space”. In IEEE conference on Computer Vision and Pat… [cited by applicant]
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. “Image synthesis with a single (robust) classifier”. In Advances in Neural Information Processing Systems, pp. 1260-… [cited by applicant]
Tsai, S. H. (2018). “Customizing an Adversarial Example Generator with Class-Conditional GANs”. [cited by applicant]
Zhao, M., Cong, Y., & Carin, L. (Aug. 2020). “On Leveraging Pretrained GANs for Generation with Limited Data”. [cited by applicant]
Lipton, Z. C., & Tripathi, S. (2017). “Precise Recovery of Latent Vectors from Generative Adversarial Networks”. [cited by applicant]
Shang, C., Palmer, A., Sun, J., Chen, K. S., Lu, J., & Bi, J. (Dec. 2017). “VIGAN: Missing View Imputation with Generative Adversarial Networks”. In 2017 IEEE International Conference on Big Data (Big Data) (pp. 766-775… [cited by applicant]
Anonymous authors. (2019). “Recycling the Discriminator for Improving the Inference Mapping of GAN”. Under review as a conference paper at ICLR 2019. Retrieved from https://openreview.net/pdf?id=HkgnpiR9Y7. [cited by applicant]
Li, A. H., Wang, Y., Chen, C., & Gao, J. (2020) “Decomposed Adversarial Learned Inference”. [cited by applicant]
Anirudh, R., Thiagarajan, J. J., Kailkhura, B., & Bremer, T. (2018). “An Unsupervised Approach to Solving Inverse Problems using Generative Adversarial Networks”. [cited by applicant]
Creswell, A., & Bharath, A.A. (2018). “Inverting the Generator of a Generative Adversarial Network”. IEE transactions on neural networks and learning systems, 30(7), 1967-1974. [cited by applicant]
Zilong Lin et al. IDSGAN: “Generative Adversarial Network for Attack Generation against Intrusion Detection,” Retrieved from the Internet: <URL: https://arxiv.org/pdf/1809.02077v3.pdf> (retrieved on Jan. 24, 2022) Jun. … [cited by applicant]
Congzheng Song et al., “Machine Leaming Models that Remember Too Much,” Retrieved from the Internet: <URL: https://arxiv.org/pdf/1709.07886.pdf>(retrieved on Jan. 24, 2022) Sep. 22, 2017(Sep. 22, 2017) section 5.2. [cited by applicant]
PCT, Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration for Application PCT/CN2021/126353 dated Jan. 29, 2022. [cited by applicant]
Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017. [cited by applicant]
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717, 2017. [cited by applicant]
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604,… [cited by applicant]
Emily L. Denton, Soumith Chintala, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems, pp. 1486-1494, 2015. [cited by applicant]
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In International Conference on Learning Representations, 2017. [cited by applicant]
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. [cited by applicant]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing … [cited by applicant]
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-… [cited by applicant]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv preprint arXiv:1912.04958, 2019. [cited by applicant]
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. https://www.cs.toronto.edu/˜kriz/cifar.html. [cited by applicant]
Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. In Advances in Neural Information Processing Systems, pp. 1328-1337, Year: 2019. [cited by applicant]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. [cited by applicant]
Japan Patent Office, “Notice of Reasons for Refusal,” Feb. 4, 2025, 8 Pages, JP Application No. 2023-532392. [cited by applicant]