IP Library Granted Patent US 10,853,577
Granted Patent B2
US 10,853,577 · App. 16/138,662 · Granted Dec 1, 2020

Response recommendation system

Inventors: Zachary Alexander (Berkeley, CA); Jayesh Govindarajan (Palo Alto, CA); Peter White (San Francisco, CA); Weiping Peng (San Francisco, CA); Colleen Smith (Mill Valley, CA); Vishal Shah (Redmond, CA); Jacob Nathaniel Huffman (Oakland, CA); Alejandro Gabriel Perez Rodriguez (Redmond, WA); Edgar Gerardo Velasco (San Francisco, CA); Na Cheng (Dublin, CA)
Assignee: salesforce.com, inc.
G06F40/30G06F40/35G06N3/0454G06N3/08G06F3/0484
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,853,577
App. No.
16/138,662
Granted
Dec 1, 2020
Kind
B2
Abstract

A data processing system analyzes a corpus of conversation data received at an interactive conversation service to train a response recommendation model. The response recommendation model generates response vectors based on custom responses and using the trained model and generates a context vector based on received input at the interactive conversation service. The context vector is compared to the set of response vectors to identify a set of recommended responses, which are recommended to an agent conversing with a user using the interactive conversation service.

Claims (68)

1. A method comprising:

receiving an input comprising a string of text at an interactive conversation service, the input received during a conversation session at the interactive conversation service;

generating, responsive to receiving the input, a context vector based at least in part on a set of inputs corresponding to the conversation session, the set of inputs including at least the received input, the context vector generated using at least one word embedding function trained on a set of training data at least partially generated by:

for a first conversation session in a corpus of conversation data, selecting a first set of sequential inputs of the first conversation session as a first context of a first context-response pair, selecting an input by an agent that is immediately subsequent to the first set of sequential inputs as a first response of the first context-response pair, and assigning a first value to a first label corresponding to the first context-response pair; and

for a second conversation session in the corpus of conversation data, selecting a second set of sequential inputs of the second conversation session as a second context of a second context-response pair, selecting a random input of the corpus conversation data as a second response of the second context-response pair, and assigning a second value to a second label corresponding to the second context-response pair;

calculating a set of similarity scores between the context vector and a set of response vectors, the set of response vectors including at least one response vector based on a custom response input by a user of the interactive conversation service; and

selecting one or more responses corresponding to one or more of the set of response vectors for input at the interactive conversation service, the one or more responses selected based on the set of similarity scores calculated between the context vector and the set of response vectors.

2. The method of claim 1 , further comprising:

generating the context vector using the at least one word embedding function trained on the corpus of conversation data corresponding to conversation sessions at instances of the interactive conversation service.

3. The method of claim 1 , further comprising:

generating the set of response vectors using the at least one word embedding function trained on the corpus of conversation data corresponding to conversation sessions at instances of the interactive conversation service.

4. The method of claim 1 , further comprising:

receiving the custom response from the user of the interactive conversation service during the conversation session.

5. The method of claim 1 , wherein the custom response is one of a set of custom responses customized for the user, a group of users, an organization, or a combination thereof.

6. The method of claim 1 , wherein the selected response is recommended to an agent for responding to the input.

7. The method of claim 1 , further comprising:

recommending the selected response to the user of the interactive conversation service.

8. The method of claim 7 , further comprising:

displaying the selected response at a user interface of a user device corresponding to the user.

9. The method of claim 1 , wherein the at least one word embedding function is further trained by:

generating a context vector for each context of each context-response pair of the set of context-response pairs using a context word embedding function of the at least one word embedding function;

generating a response vector for each response of each context-response pair of the set of context-response pairs using a response word embedding function of the at least one word embedding function;

calculating a similarity score for each context-response pair of the set of context-response pairs by comparing the context vector and the response vector of each context-response pair; and

determining a loss for each context-response pair of the set of context-response pairs based at least on the similarity score and the label corresponding to each context-response pair of the set of context-response pairs.

10. The method of claim 1 , further comprising:

receiving audio data comprising voice data at the interactive conversation service; and

converting the audio data into the input comprising the string of text.

11. A method comprising:

generating training data based on a corpus of conversation data including sequences of inputs corresponding to conversation sessions at the interactive conversation service, the training data including a set of context-response pairs and a label corresponding to each of the set of context-response pairs, a context of a context-response pair including a sequence of inputs during a conversation session, and a response being a subsequent input by an agent during the conversation session, the training data generated by:

for a first conversation session in the corpus of conversation data, selecting a first set of sequential inputs of the first conversation session as a first context of a first context-response pair, selecting an input by an agent that is immediately subsequent to the first set of sequential inputs as a first response of the first context-response pair, and assigning a first value to a first label corresponding to the first context-response pair, and

for a second conversation session in the corpus of conversation data, selecting a second set of sequential inputs of the second conversation session as a second context of a second context-response pair, selecting a random input of the corpus conversation data as a second response of the second context-response pair, and assigning a second value to a second label corresponding to the second context-response pair;

training at least one word embedding function using the training data by:

generating a context vector for each context of each context-response pair of the set of context-response pairs using a context word embedding function of the at least one word embedding function;

generating a response vector for each response of each context-response pair of the set of context-response pairs using a response word embedding function of the at least one word embedding function;

calculating a similarity score for each context-response pair of the set of context-response pairs by comparing the context vector and the response vector of each context-response pair; and

determining a loss for each context-response pair of the set of context-response pairs based at least on the similarity score and the label corresponding to each context-response pair of the set of context-response pairs.

12. The method of claim 11 , wherein the first set of sequential inputs and the second set of sequential inputs comprise a same number of inputs.

13. The method of claim 11 , wherein the context vector for each the set of context-response pairs is a vector of a first length and the response vector for each response of each context-response pair is a vector of a second length.

14. The method of claim 11 , wherein the sequences of inputs include one or more words.

15. An apparatus comprising:

a processor,

memory in electronic communication with the processor; and

instructions stored in the memory and executable by the processor to cause the apparatus to:

receive an input comprising a string of text at an interactive conversation service, the input received during a conversation session at the interactive conversation service;

generate, responsive to receiving the input, a context vector based at least in part on a set of inputs corresponding to the conversation session, the set of inputs including at least the received input, the context vector generated using at least one word embedding function trained on a set of training data at least partially generated by:

for a first conversation session in a corpus of conversation data, selecting a first set of sequential inputs of the first conversation session as a first context of a first context-response pair, selecting an input by an agent that is immediately subsequent to the first set of sequential inputs as a first response of the first context-response pair, and assigning a first value to a first label corresponding to the first context-response pair; and

for a second conversation session in the corpus of conversation data, selecting a second set of sequential inputs of the second conversation session as a second context of a second context-response pair, selecting a random input of the corpus conversation data as a second response of the second context-response pair, and assigning a second value to a second label corresponding to the second context-response pair; and

calculate a set of similarity scores between the context vector and a set of response vectors, the set of response vectors including at least one response vector based on a custom response input by a user of the interactive conversation service; and

select one or more responses corresponding to one or more of the set of response vectors for input at the interactive conversation service, the one or more responses selected based on the set of similarity scores calculated between the context vector and the set of response vectors.

16. The apparatus of claim 15 , wherein the instructions are further executable by the processor to cause the apparatus to:

generate the context vector using the at least one word embedding function trained on the corpus of conversation data corresponding to conversation sessions at instances of the interactive conversation service; and

generate the set of response vectors using the at least one word embedding function trained on the corpus of conversation data corresponding to conversation sessions at instances of the interactive conversation service.

17. The apparatus of claim 15 , wherein the selected response is recommended to an agent for responding to the input.

18. The apparatus of claim 15 , wherein the instructions are further executable by the processor to cause the apparatus to:

display the selected response at a user interface of a user device corresponding to the user for recommendation of the selected response.

19. An apparatus comprising:

a processor,

memory in electronic communication with the processor; and

instructions stored in the memory and executable by the processor to cause the apparatus to:

generate training data based on a corpus of conversation data including sequences of inputs corresponding to conversation sessions at the interactive conversation service, the training data including a set of context-response pairs and a label corresponding to each of the set of context-response pairs, a context of a context-response pair including a sequence of inputs during a conversation session, and a response being a subsequent input by an agent during the conversation session, the training data generated by:

for a first conversation session in the corpus of conversation data, selecting a first set of sequential inputs of the first conversation session as a first context of a first context-response pair, selecting an input by an agent that is immediately subsequent to the first set of sequential inputs as a first response of the first context-response pair, and assigning a first value to a first label corresponding to the first context-response pair; and

for a second conversation session in the corpus of conversation data, selecting a second set of sequential inputs of the second conversation session as a second context of a second context-response pair, selecting a random input of the corpus conversation data as a second response of the second context-response pair, and assigning a second value to a second label corresponding to the second context-response pair;

train at least one word embedding function by:

generating a context vector for each context of each context-response pair of the set of context-response pairs using a context word embedding function of the at least one word embedding function;

generating a response vector for each response of each context-response pair of the set of context-response pairs using a response word embedding function of the at least one word embedding function;

calculating a similarity score for each context-response pair of the set of context-response pairs by comparing the context vector and the response vector of each context-response pair; and

determining a loss for each context-response pair of the set of context-response pairs based at least on the similarity score and the label corresponding to each context-response pair of the set of context-response pairs.

20. The apparatus of claim 19 , wherein the first set of sequential inputs and the second set of sequential inputs comprise a same number of inputs.

Assignments (2)
CHANGE OF NAME Recorded Dec 18, 2024
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 069717/0378 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 13, 2018
From: ALEXANDER, ZACHARY; GOVINDARAJAN, JAYESH; WHITE, PETER FRANCIS; PENG, WEIPING; SMITH, COLLEEN; SHAH, VISHAL; HUFFMAN, JACOB NATHANIEL; PEREZ RODRIGUEZ, ALEJANDRO GABRIEL; VELASCO, EDGAR GERARDO; CHENG, NA
To: SALESFORCE.COM, INC.
Reel/Frame 047491/0785 →
Continuity (1)
Related Publication 20200097544A1 · Mar 26, 2020
Cited By (2)
US 12,626,705 US 12,688,196