Granted Patent B2

US 12,518,112 · App. 17/556,571 · Granted Jan 6, 2026

Inventors: Jeffry Copps Robert Jose (Tamil Nadu, IN); Reda Harb (Bellevue, WA)

Assignee: Adeia Guides Inc.

G06F40/42

Abstract

Claims (61)

1 . A method for training a natural language understanding model, the method comprising:

receiving, via a user interface of a user device, using control circuitry, a natural language query;

processing, via the user device, the natural language query using a first natural language understanding (NLU) model to generate first results, wherein the first natural language model comprises a first template of a plurality of templates to which the natural language query is mapped;

providing, via the user interface of the user device, the first results;

receiving, via the user interface of the user device, input associated with the first results;

determining, via the user device, using control circuitry, a first NLU confidence level of the understanding of the natural language query based at least in part on the received input associated with the first results wherein determining the first NLU confidence level comprises:

determining an average confidence score from a plurality of previous natural language queries;

determining a difference between the confidence level and the average confidence score;

comparing the difference between the confidence level and the average confidence score to a threshold value; and

reprocessing the natural language query when the difference between the confidence level and the average confidence score is above a threshold value; and

based at least in part on the first NLU confidence level being below an NLU confidence level threshold, reprocessing, via the user device, the natural language query using an NLU reprocessing module, wherein the reprocessing comprises:

selecting a second NLU model based on a domain of the natural language query;

generating using the NLU reprocessing model, a second template associated with the at least one second alternative result generated using the second NLU model;

based at least in part on the determining that a second NLU confidence level is greater than the first NLU confidence level, updating, via the user device, using control circuitry, the plurality of templates of the first natural language model based on the reprocessing of the natural language query; and

processing a subsequent natural language query using the updated plurality of templates of the first natural language model.

2 . The method of claim 1 , wherein the reprocessing of the natural language query comprises:

at least one of increasing a time limit for the processing the natural language query using the second natural language understanding model, or increasing a size of a dataset used for the processing the natural language query using the second natural language understanding model.

3 . The method of claim 1 , wherein the reprocessing of the natural language query using the second NLU model comprises:

processing the natural language query using rule-based natural language processing; and

determining the second NLU confidence level of the understanding of the natural language query in response to processing the natural language query using the rule-based natural language processing.

4 . The method of claim 1 , the method comprising generating a response to the natural language query, and wherein determining the first NLU confidence level comprises:

monitoring, for a predetermined period, the input into the user device subsequent to generating the response; and

determining that the input subsequent to generating the response is above an input threshold.

5 . The method of claim 1 , the method comprising generating a response to the natural language query, wherein determining the first NLU confidence level comprises:

determining metadata for an entity provided in the response;

comparing the metadata for the entity provided in the response to one or more entities of the natural language query.

6 . The method of claim 1 , the method comprising generating a response to the natural language query, wherein the determining the first NLU confidence level comprises:

determining metadata for the input associated with the first results; and

comparing the metadata of the input associated with the first results to one or more results of the natural language query.

7 . A system for training a natural language understanding model, the system comprising control circuitry configured to:

receiving, via a user interface of a user device, a natural language query;

process, via the user device, the natural language query using a first natural language understanding (NLU) model to generate first results, wherein the first natural language model comprises a first template of a plurality of templates to which the natural language query is mapped;

provide, via the user interface, the first results;

receive, via the user interface, input associated with the first results;

determine, via the user device, a first NLU confidence level of the understanding of the natural language query based at least in part on the received input associated with the first results wherein the control circuitry configured to determine the first NLU confidence level is further configured to:

determine an average confidence score from a plurality of previous natural language queries;

determine a difference between the confidence level and the average confidence score;

compare the difference between the confidence level and the average confidence score to a threshold value; and

reprocess the natural language query when the difference between the confidence level and the average confidence score is above a threshold value; and

based at least in part on the first NLU confidence level being below an NLU confidence level threshold, reprocess, via the user device, the natural language query using an NLU reprocessing module, wherein the control circuitry configured to reprocess the natural language query is further configured to:

select a second NLU model based on a domain of the natural language query;

generate using the NLU reprocessing model, a second template associated with the at least one second alternative result generated using the second NLU model;

based at least in part on the determining that a second NLU confidence level is greater than the first NLU confidence level, update, via the user device, the plurality of templates of the first natural language model based on the reprocessing of the natural language query; and

process a subsequent natural language query using the updated plurality of templates of the first natural language model.

8 . The system of claim 7 , wherein the control circuitry is configured to:

at least one of increase a time limit for the processing the natural language query using the second natural language understanding model, or increase a size of a dataset used for the processing the natural language query using the second natural language understanding model.

9 . The system of claim 7 , wherein the control circuitry is configured to:

process the natural language query using rule-based natural language processing; and

determine the second NLU confidence level of the understanding of the natural language query in response to processing the natural language query using the rule-based natural language processing.

10 . The system of claim 7 , wherein the control circuitry is configured to:

generate a response to the natural language query;

monitor, for a predetermined period, the input into the user device subsequent to generating the response; and

determine that the input subsequent to generating the response is above an input threshold.

11 . The system of claim 7 , wherein the control circuitry is configured to:

generate a response to the natural language query;

determining metadata for an entity provided in the response;

comparing the metadata for the entity provided in the response to one or more entities of the natural language query.

12 . The system of claim 7 , wherein the control circuitry is configured to:

generate a response to the natural language query;

determine metadata for the input associated with the first results; and

compare the metadata of the input associated with the first results to one or more results of the natural language query.

Assignments (3)

Continuity (1)

References Cited (44)