IP Library › Granted Patent US 12,361,221

Granted Patent B2

US 12,361,221 · App. 16/413,293 · Granted Jul 15, 2025

Grammar transfer using one or more neural networks

Inventors: Ming-Yu Liu (San Jose, CA); Kevin Lin (San Jose, CA)

Assignee: NVIDIA Corporation

G06F40/30 G06F40/253 G06N3/08 G10L15/063 G10L15/16 G10L15/1815 G10L15/22 G06N3/02

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,361,221
App. No.: 16/413,293
Granted: Jul 15, 2025
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Apparatuses, systems, and techniques to transfer grammar between sentences. In at least one embodiment, one or more first sentences are translated into one or more second sentences having different grammar using one or more neural networks.

Claims (62)

1. One or more processors, comprising:

one or more circuits to translate one or more first input portions of one or more first sentences into one or more output portions of one or more third sentences having a different style from the one or more first sentences by providing the one or more first input portions and one or more second input portions as input to one or more neural networks, the one or more second input portions comprising portions of one or more second sentences having a same style as the one or more output portions of the one or more third sentences.

2. The one or more processors of claim 1 , wherein the one or more circuits are further to:

determine, using at least one of the one or more neural networks, a content code for each of the one or more first sentences, the content code providing a style-independent expression of content of a respective first sentence.

3. The one or more processors of claim 2 , wherein the one or more circuits are further to:

determine, using at least one of the one or more neural networks, a style code to be used for the one or more third sentences, the style code being inferred from respective sentences.

4. The one or more processors of claim 3 , wherein the one or more circuits are further to:

generate latent representations of the one or more third sentences using the content code and the style codes; and

decode, using at least one decoding neural network, the latent representations into the one or more third sentences.

5. The one or more processors of claim 4 , wherein the one or more circuits are further to:

generate the latent representations using a transfer function that applies deviations of the style codes to a normalized mean of the content code.

6. The one or more processors of claim 1 , wherein the different style corresponds to a different expression of content of the one or more first sentences, the different expression differing in at least one of style, sentiment, structure, or type.

7. The one or more processors of claim 1 , wherein at least one of the one or more first sentences is received as audio data and converted into text data, or at least one of the one or more third sentences is provided as audio data generated using a digital voice corresponding to a respective style code.

8. The one or more processors of claim 1 , wherein the one or more circuits are further to:

minimize cycle-consistency loss during training of at least a subset of the one or more neural networks.

9. A system comprising:

one or more processors to translate one or more first input portions of one or more first sentences into one or more output portions of one or more third sentences having a different style from the one or more first sentences by providing the one or more first input portions and one or more second input portions as input to one or more neural networks, the one or more second input portions comprising portions of one or more second sentences having a same style as the one or more output portions of the one or more third sentences; and

one or more memories to store the one or more neural networks.

10. The system of claim 9 , wherein the one or more processors are further to:

11. The system of claim 10 , wherein the one or more processors are further to:

determine, using at least one of the one or more neural networks, a style code to be used for the one or more third sentences, the style codes being inferred from respective sentences.

12. The system of claim 11 , wherein the one or more processors are further to:

generate latent representations of the one or more third sentences using the content code and the style code; and

decode, using at least one decoding neural network, the latent representations into the one or more third sentences.

13. The system of claim 12 , wherein the one or more processors are further to:

generate the latent representations using a transfer function that applies deviations of the style codes to a normalized mean of the content code.

14. The system of claim 9 , wherein the different style corresponds to a different expression of content of the one or more first sentences, the different expression differing in at least one of style, sentiment, structure, or type.

15. The system of claim 9 , wherein at least one of the one or more first sentences is received as audio data and converted into text data, or at least one of the one or more third sentences is provided as audio data generated using a digital voice corresponding to a respective style code.

16. The system of claim 9 , wherein the one or more processors are further to:

minimize cycle-consistency loss during training of at least a subset of the one or more neural networks.

17. One or more processors, comprising:

one or more circuits to train one or more neural networks to translate one or more first input portions of one or more first sentences into one or more output portions of one or more third sentences having a different style from the one or more first sentences by providing the one or more first input portions and one or more second input portions as input to the one or more neural networks, the one or more second input portions comprising portions of one or more second sentences having a same style as the one or more output portions of the one or more third sentences.

18. The one or more processors of claim 17 , wherein the one or more circuits are further to:

19. The one or more processors of claim 18 , wherein the one or more circuits are further to:

determine, using at least one of the one or more neural networks, a style code to be used for the one or more third sentences, the style codes being inferred from respective sentences.

20. The one or more processors of claim 19 , wherein the one or more circuits are further to:

generate latent representations of the one or more third sentences using the content code and the style code; and

decode, using at least one decoding neural network, the latent representations into the one or more third sentences.

21. The one or more processors of claim 20 , wherein the one or more circuits are further to:

generate the latent representations using a transfer function that applies deviations of the style codes to a normalized mean of the content code.

22. The one or more processors of claim 17 , wherein the different style corresponds to a different expression of content of the one or more first sentences, the different expression differing in at least one of style, sentiment, structure, or type.

23. The one or more processors of claim 17 , wherein at least one of the one or more first sentences is received as audio data and converted into text data, or at least one of the one or more third sentences is provided as audio data generated using a digital voice corresponding to a respective style code.

24. The one or more processors of claim 17 , wherein the one or more circuits are further to:

minimize cycle-consistency loss during training of at least a subset of the one or more neural networks.

25. A system, comprising:

one or more processors to calculate parameters corresponding to one or more neural networks using a cycle consistency loss function applied to translating one or more first input portions of one or more first sentences into one or more output portions of one or more third sentences having a different style from the one or more first sentences by providing the one or more first input portions and one or more second input portions as input to the one or more neural networks, the one or more second input portions comprising portions of one or more second sentences having a similar style as the one or more output portions of the one or more third sentences; and

one or more memories to store the parameters.

26. The system of claim 25 , wherein the one or more processors are further to:

Determine, using at least one of the one or more neural networks, a content code for each of the one or more first sentences, the content code providing a style-independent expression of content of a respective first sentence.

27. The system of claim 26 , wherein the one or more processors are further to:

determine, using at least one of the one or more neural networks, a style code to be used for the one or more third sentences, the style codes being inferred from respective sentences.

28. The system of claim 27 , wherein the one or more processors are further to:

generate latent representations of the one or more third sentences using the content code and the style code; and

decode, using at least one decoding neural network, the latent representations into the one or more third sentences.

29. The system of claim 28 , wherein the one or more processors are further to:

generate the latent representations using a transfer function that applies deviations of the style codes to a normalized mean of the content code.

30. The system of claim 25 , wherein the different style corresponds to a different expression of content of the one or more first sentences, the different expression differing in at least one of style, sentiment, structure, or type.

31. The system of claim 25 , wherein at least one of the one or more first sentences is received as audio data and converted into text data, or at least one of the one or more third sentences is provided as audio data generated using a digital voice corresponding to a respective style code.

32. The system of claim 25 , wherein the one or more processors are further to:

minimize cycle-consistency loss during training of at least a subset of the one or more neural networks.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 13, 2019

From: LIU, MING-YU; LIN, KEVIN

To: NVIDIA CORPORATION

Reel/Frame 049457/0045 →

Continuity (1)

Related Publication 20200364303A1 · Nov 19, 2020

References Cited (61)

US 20080133245A1 · Proulx · 2008 [cited by examiner]

US 20090113289A1 · Zhang et al. · 2009 [cited by applicant]

US 20130080144A1 · Kamatani · 2013 [cited by examiner]

US 20170039174A1 · Strope · 2017 [cited by examiner]

US 20180338147A1 · Nowozin · 2018 [cited by examiner]

US 20200035220A1 · Liu · 2020 [cited by examiner]

US 20200110797A1 · Melnyk · 2020 [cited by examiner]

CN 107464554A · 2017 [cited by applicant]

CN 108304436A · 2018 [cited by applicant]

CN 109190722A · 2019 [cited by applicant]

CN 109213851A · 2019 [cited by applicant]

CN 109508457A · 2019 [cited by applicant]

CN 109635253A · 2019 [cited by applicant]

WO 2018075927 · 2018 [cited by applicant]

WO 2019052311 · 2019 [cited by applicant]

WO 2019052311A1 · 2019 [cited by applicant]

Yanpent Zhao et al., Language Style Transfer from Sentences with Arbitrary Unknown Styles, Aug. 13, 2018, Arxiv.Org, Cornell University Library, Entire Document (Year: 2018). [cited by examiner]

Extended European Search Report issued EP Application No. 20170851.8, dated Sep. 30, 2020. [cited by applicant]

Yanpeng Zhao et al: “Language Style Transfer from Sentences with Arbitrary Unknown Styles”, Arxiv.Org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Aug. 13, 2018 (Aug. 13, 2018), XP0… [cited by applicant]

Chen et al., “Unsupervised Stylish Image Description Generation via Domain Layer Norm,” Thirty-Third AAAI Conference on Artificial Intelligence, 2019, 8 pages. [cited by applicant]

Dumoulin et al., “A Learned Representation for Artistic Style,” International Conference on Learning Representations, Sep. 9, 2017, 26 pages. [cited by applicant]

Fu et al., “Style Transfer in Text: Exploration and Evaluation,” Association for the Advancement of Artificial Intelligence, 2018, 8 pages. [cited by applicant]

Gehring et al., “Convolutional Sequence to Sequence Learning,” ICML, 2017, 10 pages. [cited by applicant]

Goodfellow et al., “Generative Adversarial Nets,” In Advances in Neural Information Processing Systems, Jun. 10, 2014, 9 pages. [cited by applicant]

Guo et al., “Long Text Generation via Adversarial Training with Leaked Information,” Thirty-Second AAAI Conference on Artificial Intelligence, 2018, 8 pages. [cited by applicant]

He et al., “Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering,” Feb. 4, 2016, 11 pages. [cited by applicant]

Hu et al., “Toward Controlled Generation of Text,” Jul. 11, 2017, 10 pages. [cited by applicant]

Huang et al., “Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization,” ICCV, 2017, 10 pages. [cited by applicant]

Huang et al., “Multimodal Unsupervised Image-to-Image Translation,” In Proceedings of the European Conference on Computer Vision (ECCV), 2018, 18 pages. [cited by applicant]

Jhamtani et al., “Shakespearizing Modern Language using Copy-Enriched Sequence to Sequence Models,” EMNLP Workshop on Stylistic Variation, 2017, 10 pages. [cited by applicant]

Jones et al., “A Statistical Interpretation of Term Specificity and its Application in Retrieval,” Journal of documentation, 28(1): 1972, 9 pages. [cited by applicant]

Kerpedjiev, “Generation of Informative Texts with Style,” In International Conference on Computational Linguistics, 1992, 5 pages. [cited by applicant]

Kim et al., “Learning to Discover Cross-Domain Relations with Generative Adversarial Networks,” ICML, 2017, 9pages. [cited by applicant]

Lample et al, “Fader Networks: Manipulating Images by Sliding Attributes,” NeurIPS, Jun. 1, 2017, 10 pages. [cited by applicant]

Li et al., “Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer,” Proceedings of NAACL, 2018, 10 pages. [cited by applicant]

Lin et al., “Adversarial Ranking for Language Generation,” NeurIPS, 2017, 11 pages. [cited by applicant]

Liu et al., “Unsupervised Image-to-Image Translation Networks,” Oct. 9, 2017, 11 pages. [cited by applicant]

Mikolov et al., “Distributed Representations of Words and Phrases and their Compositionality,” NIPS, 2013, 9 pages. [cited by applicant]

Och et al., “The Alignment Template Approach to Statistical Machine Translation,” Computational Linguistics, 30(4): 2004, 33 pages. [cited by applicant]

Potash et al., “Ghostwriter: Using an LSTM for Automatic Rap Lyric Generation,” Conference on Empirical Methods in Natural Language Processing, 2015, 6 pages. [cited by applicant]

Prabhumoye et al., “Style Transfer through Back-Translation,” In Annual Meeting of the Association for Computational Linguistics, 2018, 11 pages. [cited by applicant]

Radford et al., “Learning to Generate Reviews and Discovering Sentiment,” ICLR, 2018, 11 pages. [cited by applicant]

Rao et al., “Dear Sir or Madam, May I Introduce the yafc Corpus: Corpus, Benchmarks and Metrics for Formality Style Transfer,” Proceedings of NAACL-HLT, 2018, 12 pages. [cited by applicant]

Shen et al., “Style Transfer from Non-Parallel Text by Cross-Alignment,” Nov. 6, 2017, 12 pages. [cited by applicant]

Subramanian et al., “Multiple-Attribute Text Style Transfer,” ICLR, Sep. 20, 2019, 20 pages. [cited by applicant]

Sutskever et al., “Sequence to Sequence Learning with Neural Networks,” NIPS, 2014, 9 pages. [cited by applicant]

Xu et al., “Paraphrasing for Style,” International Conference on Computational Linguistics, 2012, 16 pages. [cited by applicant]

Xu et al., “Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach,” Annual Meeting of the Association for Computational Linguistics, Jul. 2018, 10 pages. [cited by applicant]

Xu, “From Shakespeare to Twitter: What are Language Styles all about?” In Empirical Methods in Natural Language Processing Workshop, 2017, 9 pages. [cited by applicant]

Yelp, “Yelp Dataset Challenge,” retrieved on May 17, 2021, from https://web.archive.org/web/20190127144148/ https://www.yelp.com/dataset/challenge, 2019, 2 pages. [cited by applicant]

Yi et al., “DualGAN: Unsupervised Dual Learning for Image-to-Image Translation,” IEEE International Conference on Computer Vision, 2017, 9 pages. [cited by applicant]

Yu et al., “SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient,” In Association for the Advancement of Artificial Intelligence, 2017, 7 pages. [cited by applicant]

Zhang et al., “Adversarial Feature Matching for Text Generation,” Nov. 18, 2017, 10 pages. [cited by applicant]

Zhu et al., “Texygen: A Benchmarking Platform for Text Generation Models,” Feb. 6, 2018, 4 pages. [cited by applicant]

Zhu et al., “Toward Multimodal Image-to-Image Translation”, In Neural Information Processing Systems, 2017, 12 pages. [cited by applicant]

Zhu et al., “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” Nov. 24, 2017, 20 pages. [cited by applicant]

Office Action for European Application No. 20170851.8, mailed Dec. 22, 2022, 6 pages. [cited by applicant]

Zhao et al., “Language Style Transfer from Sentences with Arbitrary Unknown Styles,” Aug. 13, 2018, 12 pages. [cited by applicant]

Office Action for Chinese Application No. 202010413990.7, mailed May 15, 2023, 21 pages. [cited by applicant]

Office Action for Chinese Application No. 202010413990.7, mailed Jan. 11, 2024, 10 pages. [cited by applicant]

Office Action for Chinese Application No. 202010413990.7, mailed Apr. 28, 2024, 17 pages. [cited by applicant]

Cited By (1)

US 12,530,543