IP Library Granted Patent US 11,790,169
Granted Patent B2
US 11,790,169 · App. 17/221,691 · Granted Oct 17, 2023

Methods and systems of answering frequently asked questions (FAQs)

Inventor: Zachary Alexander (Berkeley, CA)
Assignee: Salesforce, Inc.
G06F40/279G10L15/063G10L15/22
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,790,169
App. No.
17/221,691
Granted
Oct 17, 2023
Kind
B2
Abstract

Methods and systems for answering frequently asked questions are described. An utterance is received. A decision score that is indicative of the likelihood that the utterance is answerable according to a set of frequently asked questions and associated answers is determined for the utterance. A candidate answer from the associated answers and a selection score for the candidate answer are determined for the utterance. A total score for the candidate answer is determined based on the decision score and the selection score. The total score is indicative of the likelihood that the candidate answer is a correct answer for the utterance according to the set of frequently asked questions and associated answers.

Claims (46)

1. A method performed by an answering system implemented using an electronic device, the method comprising:

receiving an utterance representative of a question from a user;

determining a decision score for the utterance that is indicative of the likelihood that the utterance is answerable according to a set of frequently asked questions and answers, wherein the determining the decision score includes:

determining a plurality of similarity scores for the utterance and the questions from the frequently asked questions and answers,

selecting a similarity score from the plurality of similarity scores for the utterance, wherein the similarity score is indicative of a question from the set of frequently asked questions and answers satisfying a similarity criterion with the utterance, and

converting the similarity score into the decision score;

determining for the utterance a candidate answer from the set of frequently asked questions and answers and a selection score for the candidate answer according to a multi-class classification model that includes the answers from the set of frequently asked questions and answers as classification classes;

determining, based on the decision score and the selection score, a total score for the candidate answer and the utterance, wherein the total score is indicative of the likelihood that the candidate answer is a correct answer for the utterance according to the set of frequently asked questions and answers; and

outputting the candidate answer as an answer to the question from the user in response to determining that the total score satisfies a threshold.

2. The method of claim 1 , wherein the determining for the utterance the candidate answer from the associated answers and the selection score for the candidate answer is based on an assumption that the question from the user is answerable according to the set of frequently asked questions and answers.

3. The method of claim 1 , wherein the multi-class classification model is trained based on a training data set that includes noisy versions of the questions and noisy versions of the answers from the set of frequently asked questions and answers.

4. The method of claim 1 , wherein the determining the total score for the candidate answer and the utterance includes:

multiplying the decision score with the selection score.

5. The method of claim 1 further comprising:

outputting the candidate answer as an answer to the question from the user and the total score as an indication of the likelihood that the candidate answer is the correct answer for the question.

6. A non-transitory machine-readable storage medium that provides instructions that, if executed by a processor, will cause said processor to perform operations comprising:

receiving an utterance representative of a question from a user;

determining a decision score for the utterance that is indicative of the likelihood that the utterance is answerable according to a set of frequently asked questions and answers, wherein the determining the decision score includes:

determining a plurality of similarity scores for the utterance and the questions from the frequently asked questions and answers,

selecting a similarity score from the plurality of similarity scores for the utterance, wherein the similarity score is indicative of a question from the set of frequently asked questions and answers satisfying a similarity criterion with the utterance, and

converting the similarity score into the decision score;

determining for the utterance a candidate answer from the set of frequently asked questions and answers and a selection score for the candidate answer according to a multi-class classification model that includes the answers from the set of frequently asked questions and answers as classification classes;

determining, based on the decision score and the selection score, a total score for the candidate answer and the utterance, wherein the total score is indicative of the likelihood that the candidate answer is a correct answer for the utterance according to the set of frequently asked questions and answers; and

outputting the candidate answer as an answer to the question from the user in response to determining that the total score satisfies a threshold.

7. The non-transitory machine-readable storage medium of claim 6 , wherein the determining for the utterance the candidate answer from the associated answers and the selection score for the candidate answer is based on an assumption that the question from the user is answerable according to the set of frequently asked questions and answers.

8. The non-transitory machine-readable storage medium of claim 6 , wherein the multi-class classification model is trained based on a training data set that includes noisy versions of the questions and noisy versions of the answers from the set of frequently asked questions and answers.

9. The non-transitory machine-readable storage medium of claim 6 , wherein the determining the total score for the candidate answer and the utterance includes:

multiplying the decision score with the selection score.

10. The non-transitory machine-readable storage medium of claim 6 , wherein the operations further comprise:

outputting the candidate answer as an answer to the question from the user and the total score as an indication of the likelihood that the candidate answer is the correct answer for the question.

11. An electronic device comprising:

a non-transitory machine-readable storage medium that provides instructions that, if executed by a processor in the electronic device, will cause the electronic device to perform operations comprising,

receiving an utterance representative of a question from a user;

determining a decision score for the utterance that is indicative of the likelihood that the utterance is answerable according to a set of frequently asked questions and answers, wherein the determining the decision score includes:

determining a plurality of similarity scores for the utterance and the questions from the frequently asked questions and answers,

selecting a similarity score from the plurality of similarity scores for the utterance, wherein the similarity score is indicative of a question from the set of frequently asked questions and answers satisfying a similarity criterion with the utterance, and

converting the similarity score into the decision score;

determining for the utterance a candidate answer from the set of frequently asked questions and answers and a selection score for the candidate answer according to a multi-class classification model that includes the answers from the set of frequently asked questions and answers as classification classes;

determining, based on the decision score and the selection score, a total score for the candidate answer and the utterance, wherein the total score is indicative of the likelihood that the candidate answer is a correct answer for the utterance according to the set of frequently asked questions and answers; and

outputting the candidate answer as an answer to the question from the user in response to determining that the total score satisfies a threshold.

12. The electronic device of claim 11 , wherein the determining for the utterance the candidate answer from the associated answers and the selection score for the candidate answer is based on an assumption that the question from the user is answerable according to the set of frequently asked questions and answers.

13. The electronic device of claim 11 , wherein the multi-class classification model is trained based on a training data set that includes noisy versions of the questions and noisy versions of the answers from the set of frequently asked questions and answers.

14. The electronic device of claim 11 , wherein the determining the total score for the candidate answer and the utterance includes:

multiplying the decision score with the selection score.

15. The electronic device of claim 11 , wherein the operations further comprise:

outputting the candidate answer as an answer to the question from the user and the total score as an indication of the likelihood that the candidate answer is the correct answer for the question.

Assignments (2)
CHANGE OF NAME Recorded Feb 17, 2023
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 062794/0656 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 2, 2021
From: ALEXANDER, ZACHARY
To: SALESFORCE.COM, INC.
Reel/Frame 055813/0518 →