IP Library › Granted Patent US 12,321,706
Granted Patent B2
US 12,321,706 · App. 18/166,806 · Granted Jun 3, 2025

Soft knowledge prompts for language models

Inventors: Siamak Shakeri (New York City, NY); Cicero Nogueira dos Santos (Glen Ridge, NJ); Daniel Matthew Cer (Santa Clara, CA); Zhe Dong (Zurich, CH); Jianmo Ni (Santa Clara, CA); Yun-Hsuan Sung (San Francisco, CA); John Nham (Fremont, CA)
Assignee: Google LLC
G06F40/295
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,321,706
App. No.
18/166,806
Granted
Jun 3, 2025
Kind
B2
Abstract

The technology employs soft knowledge prompts (KPs) to inject relevant world knowledge into language models. This includes training KPs via self-supervised learning on data from one or more knowledge bases. KPs are task independent and can function as an external memory of the language models. KPs may be entity-centric, meaning that each prompt primarily encodes information about one entity from a given knowledge base. A method includes identifying a KP in response to a received input text, concatenating that KP to a sequence of word embeddings of the input text, applying the concatenated information to a trained language model, predicting an object entity name, computing a cross-entropy loss, and updating the identified KP based on the computed cross-entropy loss.

Claims (53)

1. A computer-implemented method, comprising:

identifying, by one or more processors of a computing system, a soft knowledge prompt in response to a received input text, wherein the received input text masks an object entity;

concatenating, by the one or more processors, the identified soft knowledge prompt to a sequence of word embeddings of the input text;

applying, by the one or more processors, the concatenated soft knowledge prompt and the sequence of word embeddings to a trained language model;

predicting, by the one or more processors, an object entity name of the object entity;

computing, by the one or more processors, a cross-entropy loss according to the predicted object entity name;

updating the identified soft knowledge prompt based on the computed cross-entropy loss, wherein the updated soft knowledge prompt is configured to function as a part of an auxiliary memory of a language model that is activated when solving a specific task; and

disambiguating a named entity that appears in the received input text.

2. The computer-implemented method of claim 1 , further comprising:

selecting the updated soft knowledge prompt in response to a received query;

applying the selected soft knowledge prompt to the trained model; and

transmitting a response to the received query.

3. The computer-implemented method of claim 1 , wherein the trained language model is frozen prior to applying the concatenated soft knowledge prompt.

4. The computer-implemented method of claim 1 , wherein the soft knowledge prompt is stored in external memory separate from the trained language model.

5. The computer-implemented method of claim 4 , wherein the external memory is of variable size and a memory size of the trained language model is fixed.

6. The method of claim 1 , wherein applying the concatenated soft knowledge prompt and the sequence of word embeddings to the trained language model is done as an input to an encoder of the trained language model.

7. The method of claim 1 , wherein applying the concatenated soft knowledge prompt and the sequence of word embeddings to the trained language model is done as an input to a decoder of the trained language model.

8. The method of claim 1 , wherein updating the identified soft knowledge prompt is done by back-propagating the computed cross-entropy loss through the trained language model.

9. The method of claim 1 , wherein the cross-entropy loss is computed by applying max pooling and projection to vectors output by an encoder of the trained language model.

10. A computer-implemented method, comprising:

identifying, by one or more processors of a computing system, a soft knowledge prompt in response to a received query, the soft knowledge prompt being stored in memory external to a trained language model;

adding, by the one or more processors, the identified soft knowledge prompt to information associated with word embeddings of the received query to form a set of combined information, wherein the information associated with the word embeddings masks an object entity;

applying, by the one or more processors, the set of combined information to the trained language model;

predicting, by the one or more processors based on the set of combined information applied to the trained language model, an object entity name of the object entity;

transmitting a response to the received query according to the predicted entity name; and disambiguating a named entity that appears in the received query;

wherein the memory storing the soft knowledge prompt is configured to function as an auxiliary memory of the language model when the language model is activated to respond to the received query.

11. The method of claim 10 , wherein the soft knowledge prompt is identified by an entity linker, and the method includes applying the identification by the entity linker to the memory to retrieve the soft knowledge prompt.

12. The method of claim 10 , wherein applying the set of combined information to the trained language model is done as an input to an encoder of the trained language model.

13. The method of claim 10 , wherein applying the set of combined information to the trained language model is done as an input to a decoder of the trained language model.

14. A computing system, comprising:

memory configured to store a set of soft knowledge prompts separate from a language model; and

one or more processors operatively coupled to the memory, the one or more processors being configured to:

identify a soft knowledge prompt in response to a received input text, wherein the received input text masks an object entity;

concatenate the identified soft knowledge prompt to a sequence of word embeddings of the input text;

apply the concatenated soft knowledge prompt and the sequence of word embeddings to a trained language model;

predict an object entity name of the object entity;

compute a cross-entropy loss according to the predicted object entity name;

update, in the memory, the identified soft knowledge prompt based on the computed cross-entropy loss, wherein the updated soft knowledge prompt is configured to augment the language model when solving a specific task; and disambiguate a named entity that appears in the received input text.

15. The computing system of claim 14 , wherein application of the concatenated soft knowledge prompt and the sequence of word embeddings to the trained language model is done as an input to an encoder of the trained language model.

16. The computing system of claim 14 , wherein application of the concatenated soft knowledge prompt and the sequence of word embeddings to the trained language model is done as an input to a decoder of the trained language model.

17. The computing system of claim 14 , wherein the update of the identified soft knowledge prompt is done by back-propagating the computed cross-entropy loss through the trained language model.

18. The computing system of claim 14 , wherein the cross-entropy loss is computed by applying max pooling and projection to vectors output by an encoder of the trained language model.

19. A computing system, comprising:

memory configured to store a set of soft knowledge prompts; and

one or more processors operatively coupled to the memory, the one or more processors being configured to:

identify a soft knowledge prompt in response to a received query, the soft knowledge prompt being stored in the memory that is external to a trained language model;

add the identified soft knowledge prompt to information associated with word embeddings of the received query to form a set of combined information, wherein the information associated with the word embeddings masks an object entity;

apply the set of combined information to the trained language model;

predict, based on the set of combined information applied to the trained language model, an object entity name of the object entity;

transmit a response to the received query according to the predicted entity name; and disambiguate a named entity that appears in the received query;

wherein the memory is configured to function as an auxiliary memory of the language model when the language model is activated to respond to the received query.

20. The computing system of claim 19 , wherein application of the set of combined information to the trained language model is done as an input to an encoder of the trained language model.

21. The computing system of claim 19 , wherein application of the set of combined information to the trained language model is done as an input to a decoder of the trained language model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 14, 2023
From: SHAKERI, SIAMAK; NOGUEIRA DOS SANTOS, CICERO; CER, DANIEL MATTHEW; DONG, ZHE; NI, JIANMO; SUNG, YUN-HSUAN; NHAM, JOHN
To: GOOGLE LLC
Reel/Frame 062688/0222 →
Continuity (1)
Related Publication 20240273294A1 · Aug 15, 2024
References Cited (46)
US 20170125013A1 · Yan · 2017 [cited by examiner]
US 20220230628A1 · Zhu · 2022 [cited by examiner]
US 20230419037A1 · Chen · 2023 [cited by examiner]
US 20240070394A1 · Peng · 2024 [cited by examiner]
US 20240073159A1 · Bhatia · 2024 [cited by examiner]
US 20240211501A1 · Zhang · 2024 [cited by examiner]
CN 111639171A · 2020 [cited by applicant]
CN 114372146A · 2022 [cited by applicant]
CN 114647723A · 2022 [cited by applicant]
CN 115221331A · 2022 [cited by applicant]
CN 115438149A · 2022 [cited by applicant]
Agarwal, Oshin , et al., “Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training”, In Proceedings of the 2021 Conference of the North American Chapter of the Association for… [cited by applicant]
Bordes, Antoine , et al., “Large-scale Simple Question Answering with Memory Networks”, arXiv:1506.02075v1 [cs.LG] Jun. 5, 2015. [cited by applicant]
Dai, Damai , et al., “Knowledge Neurons in Pretrained Transformers”, arXiv:2104.08696v2 [cs.CL] Mar. 10, 2022, 10 pages. [cited by applicant]
De Jong, Michiel , et al., “Mention Memory: Incorporating Textual Knowledge Into Transformers Through Entity Mention Attention”, Published as a conference paper at ICLR 2022, 15 bages. [cited by applicant]
Devlin, Jacob , et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… [cited by applicant]
Diefenbach, Dennis , et al., “Question Answering Benchmarks for Wikidata”, In Proceedings of the ISWC 2017 Posters & Demonstrations and Industry Tracks co-located with 16th International Semantic Web Conference (ISWC 20… [cited by applicant]
Févry, Thibault , et al., “Entities as Experts: Sparse Memory Access with Entity Supervision”, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 4937-4951, Nov. 16-20, 2020. [cited by applicant]
Florez, Omar U, et al., “Aging Memories Generate More Fluent Dialogue Responses with Memory Augmented Neural Networks”, arXiv:1911.08522v2 [cs.CL] Sep. 27, 2020, 10 pages. [cited by applicant]
Guu, Kelvin , et al., “REALM: Retrieval-Augmented Language Model Pre-Training”, arXiv:2002.08909v1 [cs.CL] Feb. 10, 2020. [cited by applicant]
Hambardzumyan, Karen , et al., “WARP: Word-level Adversarial ReProgramming”, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… [cited by applicant]
Heinzerling, Benjamin , et al., “Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries”, SymarXiv:2008.09036v2 [cs.CL] Apr. 21, 2021, 22 pages. [cited by applicant]
Joshi, Mandar , et al., “TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (vol. 1: Lon… [cited by applicant]
Lester, Brian , et al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, arXiv:2104.08691v2 [cs.CL] Sep. 2, 2021, 15 pages. [cited by applicant]
Li, Xiang Lisa, et al., “Prefix-Tuning: Optimizing Continuous Prompts for Generation”, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference … [cited by applicant]
Panigrahy, Rina , et al., “Sketch based Memory for Neural Networks”, Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021, San Diego, California, USA, 12 pages. [cited by applicant]
Peters, Matthew E, et al., “Knowledge Enhanced Contextual Word Representations”, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na… [cited by applicant]
Poerner, Nina , et al., “E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT”, In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 803-818. [cited by applicant]
Radford, Alec , et al., “Language Models are Unsupervised Multitask Learners”, 2019, available at https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2… [cited by applicant]
Raffel, Colin , et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, Journal of Machine Learning Research 21 (2020), arXiv:1910.10683v3 [cs.LG] Jul. 28, 2020, pp. 1-67. [cited by applicant]
Roberts, Adam , et al., “How Much Knowledge Can You Pack Into the Parameters of a Language Model?”, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426. [cited by applicant]
Runge, Andrew , et al., “Exploring Neural Entity Representations for Semantic Information”, arXiv:2011.08951v1 [cs.CL] Nov. 17, 2020, 13 pages. [cited by applicant]
Safavi, Tara , et al., “Relational World Knowledge Representation in Contextual Language Models: A Review”, arXiv:2104.05837v2 [cs.CL] Sep. 10, 2021, 15 pages. [cited by applicant]
Sciavolino, Christopher , et al., “Simple Entity-Centric Questions Challenge Dense Retrievers”, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6138-6148. [cited by applicant]
Sukhbaatar, Sainbayar , et al., “End-To-End Memory Networks”, arXiv:1503.08895v5 [cs.NE] Nov. 24, 2015, 11 pages. [cited by applicant]
Sun, Tianxiang , et al., “CoLAKE: Contextualized Language and Knowledge Embedding”, In Proceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 3660-3670. [cited by applicant]
Taylor, Wilson L, “”Cloze Procedure“: A New Tool for Measuring Readability”, Journalism & Mass Communication Quarterly, 1953, 30:415-433. [cited by applicant]
Thorne, James , et al., “FEVER: a large-scale dataset for Fact Extraction and VERification”, In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… [cited by applicant]
Van Der Maaten, Laurens , et al., “Visualizing Data using t-SNE”, Journal of Machine Learning Research, 2008, 9(86):2579-2605. [cited by applicant]
Verga, Pat , et al., “Adaptable and Interpretable Neural Memory Over Symbolic Knowledge”, In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… [cited by applicant]
Vrandecic, Denny , et al., “Wikidata: A Free Collaborative Knowledgebase”, Communications of the ACM vol. 57, Issue 10, 2014, pp. 78-85. [cited by applicant]
Wang, Xiaozhi , et al., “Kepler: A Unified Model for Knowledge Embedding and Pre-trained Language Representation”, Transactions of the Association for Computational Linguistics, 2021, 9:176-194. [cited by applicant]
Zemlyanskiy, Yury , et al., “Readtwice: Reading Very Large Documents with Memories”, NarraarXiv:2105.04241v2 [cs.CL] May 11, 2021, 10 pages. [cited by applicant]
Zhang, Zhengyan , et al., “ERNIE: Enhanced Language Representation with Informative Entities”, In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019, pp. 1441-… [cited by applicant]
Zhang, Yuhao , et al., “Position-aware Attention and Supervised Data Improve Slot Filling”, In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 35-45, Copenhagen, Denmark. Asso… [cited by applicant]
International Search Report and Written Opinion for corresponding International Application No. PCT/US2024/012850 dated May 31, 2024 (9 pages). [cited by applicant]