Granted Patent B2

US 12,430,343 · App. 18/423,116 · Granted Sep 30, 2025

Inventors: Siddharth Jain (Mountain View, CA); Venkat Narayan Vedam (Mountain View, CA)

Assignee: Intuit Inc.

G06F16/24578G06F16/285

Abstract

Claims (293)

1. A method comprising:

receiving a query;

classifying the query to a first domain within a plurality of domains by performing operations comprising:

calculating, by a recommendation engine, a weighted confidence score for each domain as a summation of products of confidence scores of each result of a set of top results obtained from the domain, and sigmoid function values of each result of the set of top results, and a domain-specific biasing constant corresponding to each domain,

wherein the domain-specific biasing constant is calculated as a sum of products comprising a product of a current context bias weight and a current context relevance to the each domain, a product of a conversation history bias weight and a conversation history relevance to the each domain, and a popularity bias weight and a popularity of the each domain,

calculating a global confidence score, as a summation of a mean, a tunable hyperparameter, and a standard deviation, and

comparing weighted confidence scores of the plurality of domains with the global confidence score, to obtain the first domain;

determining an ambiguity associated with classifying the query;

responsive to when the ambiguity exceeds a threshold for ambiguity:

forwarding a set of ambiguous domains to a human-in-the-loop system,

receiving a ranking of the set of ambiguous domains from the human-in-the-loop system,

calculating a human-in-the-loop feedback weight from the ranking,

resolving the ambiguity by updating weighted confidence scores for the ambiguous domains based on the human-in-the-loop feedback weight, and

adjusting parameters of the recommendation engine based on the updated weighted confidence scores to cause the recommendation engine to generate modified confidence scores in response to a future user query, wherein the future user query is within a similarity threshold of the query;

responsive to when the ambiguity does not exceed the threshold for ambiguity:

retrieving an index of domain-specific vector embeddings corresponding to the first domain,

prompting a Large Language Model (LLM) with the query and the domain-specific vector embeddings, and

receiving a query response from the LLM as grounded with the domain-specific vector embeddings; and

forwarding the query response to a user.

2. The method of claim 1 , wherein comparing the plurality of domains further comprises:

if the weighted confidence score is greater than the global confidence score, retrieving results from a current context domain, and

if the weighted confidence score is not greater than the global confidence score, retrieving results from other domains.

3. A method, comprising:

receiving a query;

classifying the query to a first domain within a plurality of domains by a recommendation engine;

determining an ambiguity associated with classifying the query by performing operations comprising:

determining a global ambiguity and a focused ambiguity, and

combining the global ambiguity and the focused ambiguity to determine an overall ambiguity;

responsive to when the ambiguity exceeds a threshold for ambiguity:

forwarding a set of ambiguous domains to a human-in-the-loop system,

receiving a ranking of the set of ambiguous domains from the human-in-the-loop system,

calculating a human-in-the-loop feedback weight from the ranking,

resolving the ambiguity by updating weighted confidence scores for the ambiguous domains based on the human-in-the-loop feedback weight, and

adjusting parameters of the recommendation engine based on the updated weighted confidence scores to cause the recommendation engine to generate modified confidence scores in response to a future user query, wherein the future user query is within a similarity threshold of the query; and

responsive to when the ambiguity does not exceed the threshold for ambiguity:

retrieving an index of domain-specific vector embeddings corresponding to the first domain,

prompting a Large Language Model (LLM) with the query and the domain-specific vector embeddings,

receiving a query response from the LLM as grounded with the domain-specific vector embeddings, and

forwarding the query response to a user.

4. The method of claim 3 , wherein the global ambiguity is calculated as:

A

g

=

∑

i

=

1

d

⁢

❘

"\[LeftBracketingBar]"

global

⁢

confidence

-

weighted

⁢

confidence

i

❘

"\[RightBracketingBar]"

d

wherein:

d is a total number of domains.

5. The method of claim 3 , wherein the focused ambiguity is calculated as:

δ

=

σ

⁡

(

topN

⁢

weighted

⁢

confidence

)

max

⁡

(

weighted

⁢

confidence

)

-

min

⁡

(

weighted

⁢

confidence

)

.

6. The method of claim 3 , wherein the overall ambiguity is calculated as:

A

=

A

g

+

δ

wherein:

Ag is the global ambiguity, and

δ is the focused ambiguity.

7. A method, comprising:

receiving a query;

classifying the query to a first domain within a plurality of domains by a recommendation engine;

determining an ambiguity associated with classifying the query;

responsive to when the ambiguity exceeds a threshold for ambiguity:

forwarding a set of ambiguous domains to a human-in-the-loop system,

receiving a ranking of the set of ambiguous domains from the human-in-the-loop system,

calculating a human-in-the-loop feedback weight from the ranking for each ambiguous domain of the set of ambiguous domains,

resolving the ambiguity by updating weighted confidence scores for the set of ambiguous domains based on the human-in-the-loop feedback weight, and

adjusting parameters of the recommendation engine based on the updated weighted confidence scores to cause the recommendation engine to generate modified confidence scores in response to a future user query, wherein the future user query is within a similarity threshold of the query; and

responsive to when the ambiguity does not exceed the threshold for ambiguity:

retrieving an index of domain-specific vector embeddings corresponding to the first domain,

prompting a Large Language Model (LLM) with the query and the domain-specific vector embeddings,

receiving a query response from the LLM as grounded with the domain-specific vector embeddings, and

forwarding the query response to a user.

8. The method of claim 7 , wherein the human-in-the-loop feedback weight is calculated as:

wherein:

H

i

=

1

n

+

1

-

rank

i

n is a cardinality of the set of ambiguous domains given to the human-in-the-loop system, and

rank i is the ranking assigned to each ambiguous domain of the set of ambiguous domains by the human-in-the-loop system.

9. The method of claim 7 , wherein updating the weighted confidence score is calculated as

weighted

⁢

confidence

new

=

weighted

⁢

confidence

i

×

(

1

-

α

)

+

H

i

×

α

wherein:

H is the human-in-the-loop feedback weight, and

α is a blending factor balancing automated scoring and human feedback.

10. A system comprising:

a computer processor;

memory; and

instructions stored in the memory and executable by the computer processor to cause the computer processor to perform operations, the operations comprising:

receiving a query;

classifying the query to a first domain within a plurality of domains by a recommendation engine;

determining an ambiguity associated with classifying the query by performing operations comprising:

determining a global ambiguity and a focused ambiguity, and

combining the global ambiguity and the focused ambiguity to determine an overall ambiguity;

responsive to when the ambiguity exceeds a threshold for ambiguity:

forwarding a set of ambiguous domains to a human-in-the-loop system,

receiving a ranking of the set of ambiguous domains from the human-in-the-loop system,

calculating a human-in-the-loop feedback weight from the ranking,

resolving the ambiguity by updating weighted confidence scores for the ambiguous domains based on the human-in-the-loop feedback weight, and

adjusting parameters of the recommendation engine based on the updated weighted confidence scores to cause the recommendation engine to generate modified confidence scores in response to a future user query, wherein the future user query is within a similarity threshold of the query; and

responsive to when the ambiguity does not exceed a threshold for ambiguity:

retrieving an index of domain-specific vector embeddings corresponding to the first domain,

prompting a Large Language Model (LLM) with the query and the domain-specific vector embeddings,

receiving a query response from the LLM as grounded with the domain-specific vector embeddings, and

forwarding the query response to a user.

11. The system of claim 10 , wherein the global ambiguity is calculated as:

A

f

=

∑

i

=

1

d

⁢

❘

"\[LeftBracketingBar]"

global

⁢

confidence

-

weighted

⁢

confidence

i

❘

"\[RightBracketingBar]"

d

wherein:

d is a total number of domains,

wherein the focused ambiguity is calculated as:

δ

=

σ

⁡

(

topN

⁢

weighted

⁢

confidence

)

max

⁡

(

weighted

⁢

confidence

)

-

min

⁡

(

weighted

⁢

confidence

)

,

wherein the overall ambiguity is calculated as:

A

=

A

g

+

δ

,

and

wherein:

Ag is the global ambiguity, and

δ is the focused ambiguity.

12. The system of claim 10 , wherein the human-in-the-loop feedback weight is calculated as:

wherein:

H

i

=

1

n

+

1

-

rank

i

n is a cardinality of the set of ambiguous domains given to the human-in-the-loop system, and

rank i is the ranking assigned to each ambiguous domain of the set of ambiguous domains by the human-in-the-loop system

wherein updating the weighted confidence score is calculated as

weighted

⁢

confidence

new

=

weighted

⁢

confidence

i

×

(

1

-

α

)

+

H

i

×

α

,

and

wherein:

H is the human-in-the-loop feedback weight, and

α is a blending factor balancing automated scoring and human feedback.

13. A computer program product comprising non-transitory computer-readable program code that, when executed by a computer processor of a computing system, causes the computing system to perform operations of:

receiving a query;

classifying the query to a first domain within a plurality of domains;

determining an ambiguity associated with classifying the query;

when the ambiguity does not exceed a threshold for ambiguity:

retrieving an index of domain-specific vector embeddings corresponding to the first domain,

prompting a Large Language Model (LLM) with the query and the domain-specific vector embeddings,

receiving a query response from the LLM as grounded with the domain-specific vector embeddings, and forwarding the query response to a user; and

when the ambiguity exceeds the threshold for ambiguity:

forwarding a number of ambiguous domains to a human-in-the-loop system,

receiving a ranking of the ambiguous domains from the human-in-the-loop system,

calculating a human-in-the-loop feedback weight from the ranking, and

updating a weighted confidence score for the ambiguous domains based in the human-in-the-loop feedback weight.

Assignments (1)

Continuity (1)

References Cited (12)

Cited By (2)

US 12,675,491 US 12,711,323