IP Library Granted Patent US 11,061,955
Granted Patent B2
US 11,061,955 · App. 16/233,420 · Granted Jul 13, 2021

Intent classification system

Inventors: Zachary Alexander (Berkeley, CA); Naren M. Chittar (Saratoga, CA); Alampallam R. Ramachandran (Bellevue, WA); Anuprit Kale (Oakland, CA); Tiffany McKenzie (San Francisco, CA); Sitaram Asur (San Clara, CA); Jacob Nathaniel Huffman (Oakland, CA)
Assignee: salesforce.com, inc.
G06F16/355G06F16/3329G06F16/3347G06N20/00G06T11/206
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,061,955
App. No.
16/233,420
Granted
Jul 13, 2021
Kind
B2
Abstract

A data processing system analyzes a corpus of conversation data collected at an interactive conversation service to train an intent classification model. The intent classification model generates vectors based on the corpus of conversation data. A set of intents is selected and an intent seed input for each intent of the set of intents is input into the model to generate an intent vector corresponding to each intent. Vectors based on user inputs are generated and compared to the intent vectors to determine the intent.

Claims (78)

1. A method for intent classification, comprising:

selecting a user input from a corpus of conversation data as a context for a context-response pair for a set of context-response pairs;

selecting an agent input from the corpus of conversation data as a response for the context-response pair;

assigning a first value to a label corresponding to the context-response pair if the user input and the agent input are selected from a same conversation session of the corpus of conversation data;

assigning a second value to the label corresponding to the context-response pair if the user input and the agent input are selected from different conversation sessions of the corpus of conversation data;

training an intent classification model using the set of context-response pairs; and

implementing the intent classification model to identify an intent category for a conversation input received at an instance of an interactive conversation agent.

2. The method of claim 1 , wherein the first value is assigned further based on the user input and the agent input being selected from within a window in the same conversation session and the second value is assigned further based on the user input and the agent input being selected from outside the window in the same conversation session.

3. The method of claim 1 , wherein training the intent classification model on the set of context-response pairs further comprises:

generating a context vector for each context of the set of context-response pairs;

generating a response vector for each response of the set of context-response pairs;

calculating similarity scores for each context-response pair of the set of context-response pairs by comparing the context vector and the response vector of each context-response pair; and

determining a loss for each context-response pair of the set of context-response pairs based at least on a similarity score and the label corresponding to each context-response pair of the set of context-response pairs.

4. The method of claim 1 , further comprising:

receiving a set of intent categories and at least one intent seed input for each intent category in the set of intent categories;

generating, using a word embedding function and the intent classification model, at least one intent vector corresponding to each intent category in the set of intent categories, wherein the intent vector generated for each intent category is based at least in part on the at least one intent seed input corresponding to the respective intent category.

5. The method of claim 4 , further comprising:

receiving the conversation input comprising a text string at the instance of the interactive conversation agent;

generating, using the word embedding function and the intent classification model, a conversation input vector based on the conversation input;

calculating similarity scores between the conversation input vector and each intent vector corresponding to each intent category; and

identifying an intent category of the set of intent categories corresponding to an intent vector having a highest similarity score with the conversation input vector.

6. The method of claim 5 , wherein determining the highest similarity score further comprises:

identifying the intent vector based on a nearest neighbor technique.

7. The method of claim 5 , further comprising:

receiving audio data comprising voice data at the interactive conversation agent; and

converting the audio data into the conversation input comprising the text string.

8. The method of claim 4 , further comprising:

separating a plurality of input vectors into set of clusters based on distance score calculation between each of the plurality of input vectors; and

selecting at least one input vector from each cluster of the set of clusters, each of the selected input vectors corresponding to a suggested intent seed input, wherein one or more of the at least one intent seed input for each intent category in the set of intent categories are selected from the suggested intent seed inputs.

9. The method of claim 8 , further comprising:

transmitting an indication of each of the suggested intent seed inputs to a user device, wherein one or more of the at least one intent seed input for each intent category in the set of intent categories are selected from the suggested intent seed inputs at the user device.

10. The method of claim 4 , further comprising:

displaying a plurality of input vectors of the intent classification model on a user interface of a user device.

11. The method of claim 10 , wherein displaying the plurality of input vectors of the intent classification model further comprises:

projecting high dimensional input vectors into two-dimensional vectors;

plotting the two-dimensional vectors on a graph; and

displaying the graph at the user interface of the user device.

12. The method of claim 4 , wherein the set of intent categories includes order status, return status, return issue, product issue, return label, or a combination thereof.

13. The method of claim 4 , wherein the at least one intent seed input is based on an input received at the interactive conversation agent.

14. An apparatus for intent classification, comprising:

a processor, memory in electronic communication with the processor; and

instructions stored in the memory and executable by the processor to cause the apparatus to:

select a user input from a corpus of conversation data as a context for a context-response pair for a set of context-response pairs;

select an agent input from the corpus of conversation data as a response for the context-response pair;

assign a first value to a label corresponding to the context-response pair if the user input and the agent input are selected from a same conversation session of the corpus of conversation data;

assign a second value to the label corresponding to the context-response pair if the user input and the agent input are selected from different conversation sessions of the corpus of conversation data;

train an intent classification model using the set of context-response pairs; and

implement the intent classification model to identify an intent category for a conversation input received at an instance of an interactive conversation agent.

15. The apparatus of claim 14 , wherein the first value is assigned further based on the user input and the agent input being selected from within a window in the same conversation session and the second value is assigned further based on the user input and the agent input being selected outside the window in the same conversation session.

16. The apparatus of claim 14 , wherein the instructions to train the intent classification model on the set of context-response pairs further are executable by the processor to cause the apparatus to:

generate a context vector for each context of the set of context-response pairs;

generate a response vector for each response of the set of context-response pairs;

calculate similarity scores for each context-response pair of the set of context-response pairs by comparing the context vector and the response vector of each context-response pair; and

determine a loss for each context-response pair of the set of context-response pairs based at least on a similarity score and the label corresponding to each context-response pair of the set of context-response pairs.

17. The apparatus of claim 14 wherein the instructions are further executable by the processor to cause the apparatus to:

receive a set of intent categories and at least one intent seed input for each intent category in the set of intent categories;

generate, using a word embedding function and the intent classification model, at least one intent vector corresponding to each intent category in the set of intent categories, wherein the intent vector generated for each intent category is based at least in part on the at least one intent seed input corresponding to the respective intent category.

18. A non-transitory computer-readable medium storing code for intent classification, the code comprising instructions executable by a processor to:

select a user input from a corpus of conversation data as a context for a context-response pair for a set of context-response pairs;

select an agent input from the corpus of conversation data as a response for the context-response pair;

assign a first value to a label corresponding to the context-response pair if the user input and the agent input are selected from a same conversation session of the corpus of conversation data;

assign a second value to the label corresponding to the context-response pair if the user input and the agent input are selected from different conversation sessions of the corpus of conversation data;

train an intent classification model using the set of context-response pairs; and

implement the intent classification model to identify an intent category for a conversation input received at an instance of an interactive conversation agent.

19. The non-transitory computer-readable medium of claim 18 , wherein the first value is assigned further based on the user input and the agent input being selected from within a window in the same conversation session and the second value is assigned further based on the user input and the agent input being selected outside the window in the same conversation session.

20. The non-transitory computer-readable medium of claim 18 , wherein the instructions to train the intent classification model on the set of context-response pairs further are executable to:

generate a context vector for each context of the set of context-response pairs;

generate a response vector for each response of the set of context-response pairs;

calculate similarity scores for each context-response pair of the set of context-response pairs by comparing the context vector and the response vector of each context-response pair; and

determine a loss for each context-response pair of the set of context-response pairs based at least on a similarity score and the label corresponding to each context-response pair of the set of context-response pairs.

21. The non-transitory computer-readable medium of claim 18 , wherein the instructions are further executable to:

receive a set of intent categories and at least one intent seed input for each intent category in the set of intent categories;

generate, using a word embedding function and the intent classification model, at least one intent vector corresponding to each intent category in the set of intent categories, wherein the intent vector generated for each intent category is based at least in part on the at least one intent seed input corresponding to the respective intent category.

22. The non-transitory computer-readable medium of claim 21 , wherein the instructions are further executable to:

receive the conversation input comprising a text string at the instance of the interactive conversation agent;

generate, using the word embedding function and the intent classification model, a conversation input vector based on the conversation input;

calculate similarity scores between the conversation input vector and each intent vector corresponding to each intent category; and

identify an intent category of the set of intent categories corresponding to an intent vector having a highest similarity score with the conversation input vector.

Assignments (2)
CHANGE OF NAME Recorded Dec 18, 2024
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 069717/0394 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 14, 2019
From: ALEXANDER, ZACHARY; CHITTAR, NAREN M.; RAMACHANDRAN, ALAMPALLAM R.; KALE, ANUPRIT; MCKENZIE, TIFFANY; ASUR, SITARAM; HUFFMAN, JACOB NATHANIEL
To: SALESFORCE.COM, INC.
Reel/Frame 047985/0435 →
Continuity (2)
Continuation In Part 16138514 · Sep 21, 2018
Related Publication 20200097496A1 · Mar 26, 2020
Cited By (2)
US 12,524,610 US 12,541,496