Granted Patent B2

US 12,585,978 · App. 17/091,499 · Granted Mar 24, 2026

Inventors: Alessandro Oltramari (Pittsburgh, PA); Jonathan Francis (Pittsburgh, PA); Kaixin Ma (Pittsburgh, PA); Filip Ilievski (Marina Del Rey, CA)

Assignee: Robert Bosch GmbH

G06N20/00G06F16/9536G06N5/02

Abstract

Claims (73)

1 . A computer-implemented method for training a machine learning system, the computer-implemented method comprising:

obtaining a current data structure from a global knowledge graph that includes a combination of various knowledge graphs, the current data structure being associated with one knowledge graph of the global knowledge graph and including a current head element, a current relationship element, and a current tail element;

obtaining a sentence corresponding to the current data structure;

generating a question by removing the current tail element from the sentence;

generating a correct answer to the question, the correct answer including the current tail element;

extracting a pool of data structures from the global knowledge graph based on a set of distractor criteria, each extracted data structure having a head element with no common keywords with the current head element and a relationship element that is the current relationship element, the pool of data structures including at least one extracted data structure that is associated with another knowledge graph of the global knowledge graph;

extracting tail elements from the pool of data structures to create a pool of distractor candidates;

selecting a set of distractors from the pool of distractor candidates;

creating a query task that includes the question and a set of response options, the set of response options including the correct answer and the set of distractors;

creating a training set that includes at least the query task; and

training the machine learning system with the training set, wherein the machine learning system is configured to receive the query task and respond to the question with a predicted answer that is selected from among the set of response options.

2 . The computer-implemented method of claim 1 , wherein each extracted data structure has a tail element that is not found in another data structure of the global knowledge graph in which the another data structure includes the current head element and the current relationship element.

3 . The computer-implemented method of claim 1 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates via a random selection process; and

creating the set of distractors to include a subset of the chosen distractor candidates.

4 . The computer-implemented method of claim 1 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates that have a greatest cosine similarity score with respect to the correct answer while satisfying at least one threshold; and

creating the set of distractors to include a subset of the chosen distractor candidates.

5 . The computer-implemented method of claim 1 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates that have a greatest cosine similarity score with respect to the question while satisfying at least one threshold; and

creating the set of distractors to include a subset of the chosen distractor candidates.

6 . The computer-implemented method of claim 1 , further comprising:

obtaining a task dataset having other tasks that are distinct from the query task; and

performing a zero-shot evaluation of the machine learning system based on the task dataset,

wherein the machine learning system is trained with the training set during a pre-training phase of the machine learning system.

7 . A data processing system comprising:

at least one non-transitory computer readable medium including at least a neuro-symbolic framework, the neuro-symbolic framework including computer readable data;

a processing system including at least one processor that is operably connected to the at least one non-transitory computer readable medium, the processor being configured to execute the computer readable data to implement a method that includes:

obtaining a current data structure from a global knowledge graph that includes a combination of various knowledge graphs, the current data structure being associated with one knowledge graph of the global knowledge graph and including a current head element, a current relationship element, and a current tail element;

obtaining a sentence corresponding to the current data structure;

generating a question by removing the current tail element from the sentence;

generating a correct answer to the question, the correct answer including the current tail element;

extracting a pool of data structures from the global knowledge graph based on a set of distractor criteria, each extracted data structure having a head element with no common keywords with the current head element and a relationship element that is the current relationship element, the pool of data structures including at least one extracted data structure that is associated with another knowledge graph of the global knowledge graph;

extracting tail elements from the pool of data structures to create a pool of distractor candidates;

selecting a set of distractors from the pool of distractor candidates;

creating a query task that includes the question and a set of response options, the set of response options including the correct answer and the set of distractors;

creating a training set that includes at least the query task; and

training a machine learning system with the training set, wherein the machine learning system is configured to receive the query task and respond to the question with a predicted answer that is selected from among the set of response options.

8 . The data processing system of claim 7 , wherein each extracted data structure has a tail element that is not found in another data structure of the global knowledge graph in which the another data structure includes the current head element and the current relationship element.

9 . The data processing system of claim 7 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates via a random selection process; and

creating the set of distractors to include a subset of the chosen distractor candidates.

10 . The data processing system of claim 7 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates that have a greatest cosine similarity score with respect to the correct answer while satisfying at least one threshold; and

creating the set of distractors to include a subset of the chosen distractor candidates.

11 . The data processing system of claim 7 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates that have a greatest cosine similarity score with respect to the question while satisfying at least one threshold; and

creating the set of distractors to include a subset of the chosen distractor candidates.

12 . The data processing system of claim 7 , wherein the method that further comprises:

obtaining a task dataset having other tasks that are distinct from the query task; and

performing a zero-shot evaluation of the machine learning system based on the task dataset,

wherein the machine learning system is trained with the training set during a pre-training phase of the machine learning system.

13 . A computer product comprising at least one non-transitory computer readable storage device that includes computer-readable data, which when executed by one or more processors, is operable to cause the one or more processors to implement a method that comprises:

obtaining a current data structure from a global knowledge graph that includes a combination of various knowledge graphs, the current data structure being associated with one knowledge graph of the global knowledge graph and including a current head element, a current relationship element, and a current tail element;

obtaining a sentence corresponding to the current data structure;

generating a question by removing the current tail element from the sentence;

generating a correct answer to the question, the correct answer including the current tail element;

extracting a pool of data structures from the global knowledge graph based on a set of distractor criteria, each extracted data structure having a head element with no common keywords with the current head element and a relationship element that is the current relationship element, the pool of data structures including at least one extracted data structure that is associated with another knowledge graph of the global knowledge graph;

extracting tail elements from the pool of data structures to create a pool of distractor candidates;

selecting a set of distractors from the pool of distractor candidates;

creating a query task that includes the question and a set of response options, the set of response options including the correct answer and the set of distractors;

creating a training set that includes at least the query task; and

training a machine learning system with the training set, wherein the machine learning system is configured to receive the query task and respond to the question with a predicted answer that is selected from among the set of response options.

14 . The computer product of claim 13 , wherein each extracted data structure has a tail element that is not found in another data structure of the global knowledge graph in which the another data structure includes the current head element and the current relationship element.

15 . The computer product of claim 13 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates via a random selection process; and

creating the set of distractors to include a subset of the chosen distractor candidates.

16 . The computer product of claim 13 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates that have a greatest cosine similarity score with respect to the correct answer while satisfying at least one threshold; and

creating the set of distractors to include a subset of the chosen distractor candidates.

17 . The computer product of claim 13 , wherein the step of selecting the set of distractors from the pool of distractor candidates comprises:

choosing distractor candidates that have a greatest cosine similarity score with respect to the question while satisfying at least one threshold; and

creating the set of distractors to include a subset of the chosen distractor candidates.

Assignments (1)

Continuity (1)

References Cited (84)