IP Library › Granted Patent US 12,205,026
Granted Patent B2
US 12,205,026 · App. 17/674,461 · Granted Jan 21, 2025

Enhanced word embedding

Inventor: Christopher Malon (Fort Lee, NJ)
Assignee: NEC Corporation
G06N3/08G06F40/166G06F40/242G06F40/279
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,205,026
App. No.
17/674,461
Granted
Jan 21, 2025
Kind
B2
Abstract

Methods and systems for language processing include augmenting an original training dataset to produce an augmented dataset that includes a first example that includes a first scrambled replacement for a first word and a definition of the first word, and a second example that includes a second scrambled replacement for the first word and a definition of an alternative to the first word. A neural network classifier is trained using the augmented dataset.

Claims (24)

1. A method for language processing, comprising:

augmenting an original training dataset to produce an augmented dataset that includes a first example that includes a first scrambled replacement for a first word and a definition of the first word, and a second example that includes a second scrambled replacement for the first word and a definition of an alternative to the first word, wherein the first example and an original example have a same label and the second example has a different label; and

training a neural network classifier using the augmented dataset.

2. The method of claim 1 , wherein augmenting an original dataset includes replacing the original example with first example and the second example.

3. The method of claim 2 , wherein augmenting the original dataset includes selecting the first word from the original example by determining that the alternative to the first word has an above-threshold probability.

4. The method of claim 3 , wherein selecting the first word from the first example further includes determining that the first word only occurs once in the first example.

5. The method of claim 2 , wherein the first scrambled word occupies a same place in the first example as the first word occupies in the original example.

6. The method of claim 1 , wherein the labels of the second example are determined by a classifier that is trained using the original example, but not the first example or second example.

7. The method of claim 1 , wherein the scrambled word is a string of random characters.

8. The method of claim 1 , further comprising performing a natural language processing task on input text using the trained neural network classifier.

9. The method of claim 8 , wherein the natural language processing task is selected from the group consisting of natural language inference, fact verification, opinion analysis, sentiment analysis, and topic classification.

10. A system for language processing, comprising:

a hardware processor; and

a memory that stores a computer program, which, when executed by the hardware processor, causes the hardware processor to:

augment an original training dataset to produce an augmented dataset that includes a first example that includes a first scrambled replacement for a first word and a definition of the first word, and a second example that includes a second scrambled replacement for the first word and a definition of an alternative to the first word, wherein the first example and an original example have a same label and the second example has a different label; and

train a neural network classifier using the augmented dataset.

11. The system of claim 10 , wherein the computer program further causes the hardware processor to replace the original example with first example and the second example.

12. The system of claim 11 , wherein the computer program further causes the hardware processor to select the first word from the original example by determining that the alternative to the first word has an above-threshold probability.

13. The system of claim 12 , wherein the computer program further causes the hardware processor to determine that the first word only occurs once in the first example.

14. The system of claim 11 , wherein the first scrambled word occupies a same place in the first example as the first word occupies in the original example.

15. The system of claim 10 , wherein the labels of the second example are determined by a classifier that is trained using the original example, but not the first example or second example.

16. The system of claim 10 , wherein the scrambled word is a string of random characters.

17. The system of claim 10 , wherein the computer program further causes the hardware processor to perform a natural language processing task on input text using the trained neural network classifier.

18. The system of claim 17 , wherein the natural language processing task is selected from the group consisting of natural language inference, fact verification, opinion analysis, sentiment analysis, and topic classification.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 10, 2024
From: NEC LABORATORIES AMERICA, INC.
To: NEC CORPORATION
Reel/Frame 069540/0269 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 17, 2022
From: MALON, CHRISTOPHER
To: NEC LABORATORIES AMERICA, INC.
Reel/Frame 059038/0461 →
Continuity (2)
Provisional Application 63154893 · Mar 1, 2021
Related Publication 20220277197A1 · Sep 1, 2022
References Cited (22)
US 11461668B1 · Mohandas · 2022 [cited by examiner]
US 11657307B1 · Bodapati · 2023 [cited by examiner]
US 20070136048A1 · Richardson-Bunbury · 2007 [cited by examiner]
US 20220050967A1 · Veyseh · 2022 [cited by examiner]
Liu, Yinhan, et al. “ROBERTa: A Robustly Optimized BERT Pretraining Approach”, arXiv:1907.11692v1 [cs.CL]. Jul. 26, 2019, pp. 1-13. [cited by applicant]
Lauscher, Anne, et al. “Common Sense orWorld Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers”, Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Ex… [cited by applicant]
Schick, Timo, et al. “BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Jul. 2020, pp. 3996-… [cited by applicant]
Nie, Yixin, et al. “Adversarial NLI: A New Benchmark for Natural Language Understanding”, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Jul. 2020, pp. 4885-4901. [cited by applicant]
Kim, Siwon, et al. “Interpretation of NLP models through input marginalization”, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Nov. 2020, pp. 3154-3167. [cited by applicant]
Zhu, Yukun, et al. “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books”, InProceedings of the IEEE international conference on computer vision. Dec. 7, 2015, pp. 19-27. [cited by applicant]
Devlin, Jacob, et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, arXiv:1810.04805v2 [cs.CL]. May 24, 2019, pp. 1-6. [cited by applicant]
Speer, Robyn, et al. “ConceptNet 5.5: An Open Multilingual Graph of General Knowledge”, arXiv:1612.03975v2 [cs.CL]. Dec. 11, 2018, pp. 1-9. [cited by applicant]
Mikolov, Tomas, et al. “Efficient Estimation of Word Representations in Vector Space”, arXiv:1301.3781v3 [cs.CL]. Sep. 7, 2013, pp. 1-12. [cited by applicant]
Kaneko, Masahiro, et al. “Dictionary-based Debiasing of Pre-trainedWord Embeddings”, prearXiv:2101.09525v1 [cs.CL]. Jan. 23, 2021, pp. 1-12. [cited by applicant]
Kudo, Taku, et al. “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing”, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing … [cited by applicant]
De Smedt, Tom, et al. “Pattern for Python”, Journal of Machine Learning Research 13. Jun. 2012, pp. 2063-2067. [cited by applicant]
Yang, Zhilin, et al. “XInet: Generalized autoregressive pretraining for language understanding”, Advances in neural information processing systems, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019)… [cited by applicant]
Singh, Push, et al. “Open mind common sense: Knowledge acquisition from the general public”, InOTM Confederated International Conferences On the Move to Meaningful Internet Systems, Springer, Berlin, Heidelberg. Oct. 30… [cited by applicant]
Talmor, Alon, et al. “Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge”, 34th Conference on Neural Information Processing Systems (NeurIPS 2020). Jun. 11, 2020, pp. 1-11. [cited by applicant]
Bowman, Samuel R., et al. “A large annotated corpus for learning natural language inference”, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal. Sep. 2015, pp. 632-… [cited by applicant]
Tissier, Julien, et al. “Dict2vec : LearningWord Embeddings using Lexical Dictionaries”, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark. Sep. 2017, pp. 254-26… [cited by applicant]
Chen, Qian, et al. “Neural Natural Language Inference Models Enhanced with External Knowledge”, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), Melbourne, Australia… [cited by applicant]