IP Library › Granted Patent US 12,488,194
Granted Patent B2
US 12,488,194 · App. 17/707,080 · Granted Dec 2, 2025

System and method with entity type clarification for fine-grained factual knowledge retrieval

Inventor: Jun Araki (San Jose, CA)
Assignee: Robert Bosch GmbH
G06F40/40G06F40/295G06F40/35G06N5/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,488,194
App. No.
17/707,080
Filed
Mar 29, 2022
Granted
Dec 2, 2025
Kind
B2
Art Unit
2657
USPC
704/9
Abstract

A computer-implemented system and method relate to factual knowledge retrieval with entity type clarification. A set of candidates is generated for a first prompt. The set of candidates provide a solution to the first prompt. A set of second prompts is generated based on the set of candidates. A set of entity types is generated using the set of second prompts. The set of entity types categorize the set of candidates. The set of entity types is output via a user interface. A selected entity type is received via the user interface. The selected entity type is chosen from among the set of entity types. A selected candidate is output. The selected candidate corresponds to the selected entity type.

Claims (74)

1 . A computer-implemented method for a virtual assistant with factual knowledge retrieval and entity type clarification, the method comprising:

obtaining a first prompt that includes an input entity and a relation;

generating, via a machine learning system, a set of candidates based on the input entity and the relation of the first prompt, each candidate being a target entity that is predicted as a next word based on the input entity and the relation, the machine learning system including an autoregressive language model that is pretrained;

generating a set of second prompts by applying a template to the set of candidates, each second prompt including a respective candidate and an entity type slot to receive an entity type of the respective candidate;

generating, via the autoregressive language model, a set of entity types using the set of second prompts, the set of entity types categorizing the set of candidates, each entity type having a greatest confidence score with respect to satisfying the entity type slot of the respective second prompt;

outputting, via a user interface, the set of entity types;

receiving, via the user interface, a selected entity type that is chosen from among the set of entity types to obtain output data of a desired scope for the first prompt;

providing an answer candidate as the output data, the answer candidate being selected from among the set of candidates based on the selected entity type;

generating a knowledge graph that includes at least a semantic triple, the semantic triple including the input entity, the relation, and the output data; and

using, via the virtual assistant, the semantic triple in a question-answering system.

2 . The computer-implemented method of claim 1 ,

wherein,

the first prompt is generated based on the input entity using another template;

the another template includes an input slot to receive the input entity;

the another template includes an answer slot that provides a solution to the input slot;

the machine learning system is configured to generate a first candidate as a first answer that satisfies the answer slot when the input entity is in the input slot; and

the set of candidates includes the first candidate.

3 . The computer-implemented method of claim 1 , further comprising:

generating, via the machine learning system, a set of confidence scores for the set of candidates,

wherein,

the answer candidate is associated with a particular confidence score,

the set of candidates include another candidate,

the another candidate is associated with another confidence score, and

the particular confidence score is less than the another confidence score.

4 . The computer-implemented method of claim 1 , further comprising:

updating a knowledge base with a data structure that includes the first prompt and the answer candidate,

wherein the data structure includes a semantic triple that is stored in a non-transitory computer-readable medium.

5 . A computer system for knowledge retrieval with interactive entity type clarification, the computer system comprising:

a user interface;

a non-transitory computer readable medium storing computer readable data including instructions that, when executed, perform a method; and

a processor in data communication with the user interface and the non-transitory computer readable medium, the processor being configured to execute the computer readable data to perform the method that includes:

obtaining a first prompt that includes an input entity and a relation;

generating, via a machine learning system, a set of candidates based on the input entity and the relation of the first prompt, each candidate being a target entity that is predicted as a next word based on the input entity and the relation, the machine learning system including an autoregressive language model that is pretrained;

generating a set of second prompts by applying a template to the set of candidates, each second prompt including a respective candidate and an entity type slot to receive an entity type of the respective candidate;

generating, via the autoregressive language model, a set of entity types using the set of second prompts, the set of entity types categorizing the set of candidates, each entity type having a greatest confidence score with respect to satisfying the entity slot of the respective second prompt;

outputting, via the user interface, the set of entity types;

receiving, via the user interface, a selected entity type that is chosen from among the set of entity types to obtain output data of a desired scope for the first prompt;

providing an answer candidate as the output data, the answer candidate being selected from among the set of candidates based on the selected entity type;

generating a knowledge graph that includes at least a semantic triple, the semantic triple including the input entity, the relation, and the output data; and

using, via the virtual assistant, the semantic triple in a question-answering system of the virtual assistant technology, the selected answer candidate corresponding to the selected answer entity type.

6 . The computer system of claim 5 ,

wherein,

the first prompt is generated based on the input entity using another template;

the another template includes an input slot to receive the input entity;

the another template includes an answer slot that provides a solution to the input slot;

the machine learning system is configured to generate a first candidate as a first answer that satisfies the answer slot when the input entity is in the input slot; and

the set of candidates includes the first candidate.

7 . The computer system of claim 5 , further comprising:

generating, via the machine learning system, a set of confidence scores for the set of candidates,

wherein,

the answer candidate is associated with a particular confidence score,

the set of candidates include another candidate,

the another candidate is associated with another confidence score, and

the particular confidence score is less than the another confidence score.

8 . The computer system of claim 5 , further comprising:

updating a knowledge base with a data structure that includes the first prompt and the selected candidate,

wherein the data structure includes a semantic triple that is stored in the non-transitory computer readable medium.

9 . One or more non-transitory computer readable storage media storing computer readable data with instructions that when executed by one or more processors cause the one or more processors to perform a method for a virtual assistant that comprises:

obtaining a first prompt that includes an input entity and a relation;

generating, via a machine learning system, a set of candidates based on the input entity and the relation of the first prompt, each candidate being a target entity that is predicted as a next word based on the input entity and the relation, the machine learning system including an autoregressive language model that is pretrained;

generating a set of second prompts by applying a template to the set of candidates, each second prompt including a respective candidate and an entity type slot to receive an entity type of the respective candidate;

generating, via the autoregressive language model, a set of entity types using the set of second prompts, the set of entity types categorizing the set of candidates each entity type having a greatest confidence score with respect to satisfying the entity slot of the respective second prompt;

outputting, via a user interface, the set of entity types;

receiving, via the user interface, a selected entity type that is chosen from among the set of entity types to obtain output data of a desired scope for the first prompt;

providing an answer candidate as the output data, the answer candidate being selected from among the set of candidates based on the selected entity type;

generating a knowledge graph that includes at least a semantic triple, the semantic triple including the input entity, the relation, and the output data; and

using, via the virtual assistant, the semantic triple in a question-answering system.

10 . The one or more non-transitory computer readable storage media of claim 9 , further comprising:

generating, via the machine learning system, a set of confidence scores for the set of candidates,

wherein,

the answer candidate is associated with a particular confidence score,

the set of candidates include another candidate,

the another candidate is associated with another confidence score, and

the particular confidence score is less than the another confidence score.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 2, 2022
From: ARAKI, JUN
To: ROBERT BOSCH GMBH
Reel/Frame 059781/0902 →
Continuity (1)
Related Publication 20230316001A1 · Oct 5, 2023
References Cited (41)
US 9466294B1 · Tunstall-Pedoe · 2016 [cited by examiner]
US 20180233141A1 · Solomon · 2018 [cited by examiner]
US 20200151607A1 · Bishop · 2020 [cited by examiner]
US 20220147835A1 · Kuo · 2022 [cited by examiner]
US 20220414129A1 · Kurumaddali · 2022 [cited by examiner]
US 20230076127A1 · Yu · 2023 [cited by examiner]
US 20230177363A1 · Li · 2023 [cited by examiner]
US 20230205824A1 · Jablokov · 2023 [cited by examiner]
US 20230237277A1 · Reza · 2023 [cited by examiner]
De Cao, Nicola, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. “Autoregressive entity retrieval.” arXiv preprint arXiv:2010.00904 (2020). (Year: 2020). [cited by examiner]
Anderson et al., Guided Open Vocabulary Image Captioning with Constrained Beam Search, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, Sep. 7-11, 2017, Associ… [cited by applicant]
Araki et al., Open-Domain Event Detection using Distant Supervision, Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, Association for Computational Linguistics, A… [cited by applicant]
Brown et al., Language Models are Few-Shot Learners, Proceedings of Advances in Neural Information Processing Systems, 2020, pp. 1877-1901. [cited by applicant]
Cao et al., Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer… [cited by applicant]
Dathathri et al., Plug and Play Language Models: A Simple Approach to Controlled Text Generation, Proceedings of the International Conference on Learning Representations, 2020, pp. 1-34. [cited by applicant]
Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… [cited by applicant]
Ficler et al., Controlling Linguistic Style Aspects in Neural Language Generation, Proceedings of the Workshop on Stylistic Variation, Copenhagen, Denmark, Association for Computational Linguistics, Sep. 7-11, 2017, pp.… [cited by applicant]
Haviv et al., BERTese: Learning to speak to BERT, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Online, Association for Computational Linguistics, Apr. 19-2… [cited by applicant]
Hokamp et al., Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long Papers), Vancouver, Can… [cited by applicant]
Hu et al., Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… [cited by applicant]
Jiang et al., X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 16-20, 2020, Associat… [cited by applicant]
Jiang et al., How Can We Know What Language Models Know?, Transactions of the Association for Computational Linguistics, vol. 8, 2020, pp. 423-438. [cited by applicant]
Keskar et al., CTRL: A Conditional Transformer Language Model for Controllable Generation, arXiv:1909.05858v2, Sep. 20, 2019, pp. 1-18. [cited by applicant]
Kikuchi et al., Controlling Output Length in Neural Encoder-Decoders, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, Nov. 1-5, 2016, pp. 1328-1338. [cited by applicant]
Liu et al., Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, arXiv:2107.13586v1, Jul. 28, 2021, pp. 1-46. [cited by applicant]
Liu et al., GPT Understands, Too, arXiv:2103.10385v1, Mar. 18, 2021, pp. 1-10. [cited by applicant]
Lu et al., NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: … [cited by applicant]
Miao et al., CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling, Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), Association for the Advancement of Artificial Int… [cited by applicant]
Pascual et al., Directed Beam Search: Plug-and-Play Lexically Constrained Language Generation arXiv:2012.15416v1, Dec. 31, 2020, pp. 1-9. [cited by applicant]
Perez et al., True Few-Shot Learning with Language Models, arXiv:2105.11447v1, May 24, 2021, pp. 1-21. [cited by applicant]
Petroni et al., How Context Affects Language Models' Factual Predictions, Proceedings of the Conference on Automated Knowledge Base Construction, 2020, pp. 1-15. [cited by applicant]
Petroni et al., Language Models as knowledge bases?, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong… [cited by applicant]
Post et al., Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics:… [cited by applicant]
Qin et al., Learning How to Ask: Querying LMs with Mixtures of Soft Prompts, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 202… [cited by applicant]
Salazar et al., Masked Language Model Scoring, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, Association for Computational Linguistics, pp. 2699-2712. [cited by applicant]
Sha, Gradient-guided Unsupervised Lexically Constrained Text Generation, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 16-20, 2020, Association for Computational Li… [cited by applicant]
Shin et al., Autoprompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, Association f… [cited by applicant]
Vaswani et al., Attention is All You Need, Advances in Neural Information Processing Systems, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, pp. 1-11. [cited by applicant]
Yu et al., SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, Association for the Advancement of Artificial Intelligence,… [cited by applicant]
Zhong et al., Factual Probing is [MASK]: Learning vs. Learning to Recall, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2… [cited by applicant]
Ziegler et al., Fine-Tuning Language Models from Human Preferences, arXiv:1909.08593v2, Jan. 8, 2020, pp. 1-26. [cited by applicant]
Cited By (1)
US 12,647,326