Descriptive topic modeling with LDA on bags of utterance clusters
The present disclosure relates to a system, a method, and a product for intent discovery. The system includes a processor in communication with a memory storing instructions. When the processor executes the instructions, the instructions are configured to cause the processor to: obtain documents comprising a set of utterances, extract the set of utterances from the documents, generate a set of utterance embeddings based on the set of utterances, clusterize the set of utterance embeddings to obtain a plurality of clusters, obtain a cluster label for each cluster, encode each document based on a number of times each utterance cluster identifier (ID) appears to obtain an encoded document, perform latent Dirichlet allocation (LDA) on the encoded documents to obtain K topics, and each topic corresponding to a list of key clusters with cluster IDs, and for each topic, replace the cluster IDs with the cluster labels.
1. A system for intent discovery, the system comprising:
a memory storing instructions; and
a processor in communication with the memory, wherein, when the processor executes the instructions, the instructions are configured to cause the processor to:
obtain documents comprising a set of utterances;
extract the set of utterances from the documents;
generate a set of utterance embeddings based on the set of utterances;
clusterize the set of utterance embeddings to obtain a plurality of clusters, each cluster comprising one or more utterance embeddings;
obtain a cluster label for each cluster, wherein to obtain the cluster label for each cluster, the instructions are further configured to cause the processor to:
calculate a center in an embedding space for the one or more utterance embeddings in a corresponding cluster of the plurality of clusters;
select a central utterance embedding in the corresponding cluster, the central utterance embedding being a closest utterance embedding to the center among the one or more utterance embeddings in the corresponding cluster; and
obtain an utterance corresponding to the central utterance embedding as a corresponding cluster label of the corresponding cluster;
encode each document of the documents based on a number of times each utterance cluster identifier (ID) appears to obtain an encoded document, the encoded document comprising at least one cluster ID with a weight;
perform latent Dirichlet allocation (LDA) on the encoded documents to obtain K topics, wherein K is a positive integer, and each topic of the K topics corresponds to a list of key clusters with cluster IDs; and
for each topic, replace the cluster IDs with corresponding cluster labels.
2. The system of claim 1 , wherein to generate the set of utterance embeddings based on the set of utterances, the instructions are further configured to cause the processor to:
generate the set of utterance embeddings based on the set of utterances with a neural network sentence encoder comprising one of the following models:
a sentence bidirectional encoder representation from transformers (SBERT) sentence encoder,
a universal sentence encoder (USE), and
a multilingual universal sentence encoder (MUSE).
3. The system of claim 1 , wherein to clusterize the set of utterance embeddings to obtain the plurality of clusters, the instructions are further configured to cause the processor to:
clusterize, based on a first hyperparameter and a second hyperparameter, the set of utterance embeddings to obtain the plurality of clusters, the first hyperparameter indicating a minimum number of utterance embeddings in any cluster, the second hyperparameter indicating a maximum distance between any neighboring pair of utterance embeddings in any cluster.
4. The system of claim 1 , wherein to clusterize the set of utterance embeddings to obtain the plurality of clusters, the instructions are further configured to cause the processor to:
clusterize the set of utterance embeddings to obtain the plurality of clusters with a clustering algorithm of density-based spatial clustering of applications with noise (DBSCAN).
5. The system of claim 1 , wherein to clusterize the set of utterance embeddings to obtain the plurality of clusters, the instructions are further configured to cause the processor to:
partition the set of utterance embeddings into a plurality of subsets of utterance embeddings;
for each subset, clusterize the subset of utterance embeddings to obtain a subset of clusters; and
combine all subsets of clusters to form the plurality of clusters.
6. The system of claim 1 , wherein:
the number of times each utterance in the cluster appearing in the documents is used to calculate a term frequency of a cluster ID corresponding to the utterance and an inverse document frequency of the cluster ID corresponding to the utterance.
7. The system of claim 1 , wherein the instructions are further configured to cause the processor to:
determine a value of the K using an exponential decay method.
8. A method for intent discovery, the method comprising:
obtaining, by a device comprising a memory storing instructions and a processor in communication with the memory, documents comprising a set of utterances;
extracting, by the device, the set of utterances from the documents;
generating, by the device, a set of utterance embeddings based on the set of utterances;
clusterizing, by the device, the set of utterance embeddings to obtain a plurality of clusters, wherein each cluster comprises one or more utterance embeddings;
obtaining, by the device, a cluster label for each cluster by:
calculating a center in an embedding space for one or more utterance embeddings in a corresponding cluster of the plurality of clusters,
selecting a central utterance embedding in the corresponding cluster, the central utterance embedding being a closest utterance embedding to the center among the one or more utterance embeddings in the corresponding cluster, and
obtaining an utterance corresponding to the central utterance embedding as a corresponding cluster label of the corresponding cluster;
encoding, by the device, each document of the documents based on a number of times each utterance cluster identifier (ID) appears to obtain an encoded document, wherein the encoded document comprises at least one cluster ID with a weight;
performing, by the device, latent Dirichlet allocation (LDA) on the encoded documents to obtain K topics, wherein K is a positive integer, and each topic of the K topics corresponds to a list of key clusters with cluster IDs; and
for each topic, replacing, by the device, the cluster IDs with corresponding cluster labels.
9. The method of claim 8 , wherein the generating the set of utterance embeddings based on the set of utterances comprises:
generating the set of utterance embeddings based on the set of utterances with a neural network sentence encoder comprising one of the following models:
a sentence bidirectional encoder representation from transformers (SBERT) sentence encoder,
a universal sentence encoder (USE), and
a multilingual universal sentence encoder (MUSE).
10. The method of claim 8 , wherein the clusterizing the set of utterance embeddings to obtain the plurality of clusters comprises:
clusterizing, based on a first hyperparameter and a second hyperparameter, the set of utterance embeddings to obtain the plurality of clusters, the first hyperparameter indicating a minimum number of utterance embeddings in any cluster, the second hyperparameter indicating a maximum distance between any neighboring pair of utterance embeddings in any cluster.
11. The method of claim 8 , wherein the clusterizing the set of utterance embeddings to obtain the plurality of clusters comprises:
clusterizing the set of utterance embeddings to obtain the plurality of clusters with a clustering algorithm of density-based spatial clustering of applications with noise (DBSCAN).
12. The method of claim 8 , wherein the clusterizing the set of utterance embeddings to obtain the plurality of clusters comprises:
partitioning the set of utterance embeddings into a plurality of subsets of utterance embeddings,
for each subset, clusterizing the subset of utterance embeddings to obtain a subset of clusters, and
combining all subsets of clusters to form the plurality of clusters.
13. The method of claim 8 , wherein:
the number of times each utterance in the cluster appearing in the documents is used to calculate a term frequency of a cluster ID corresponding to the utterance and an inverse document frequency of the cluster ID corresponding to the utterance.
14. The method of claim 8 , further comprising:
determining a value of the K using an exponential decay method.
15. A non-transitory computer-readable medium (CRM) storing a set of instructions for intent discovery,
wherein the set of instructions, when executed by a processor of a system, cause the processor to:
obtain documents comprising a set of utterances;
extract the set of utterances from the documents;
generate a set of utterance embeddings based on the set of utterances;
clusterize the set of utterance embeddings to obtain a plurality of clusters, each cluster comprising one or more utterance embeddings;
obtain a cluster label for each cluster, wherein to obtain the cluster label for each cluster, the instructions are further configured to cause the processor to:
calculate a center in an embedding space for the one or more utterance embeddings in a corresponding cluster of the plurality of clusters;
select a central utterance embedding in the corresponding cluster, the central utterance embedding being a closest utterance embedding to the center among the one or more utterance embeddings in the corresponding cluster; and
obtain an utterance corresponding to the central utterance embedding as a corresponding cluster label of the corresponding cluster;
encode each document of the documents based on a number of times each utterance cluster identifier (ID) appears to obtain an encoded document, the encoded document comprising at least one cluster ID with a weight;
perform latent Dirichlet allocation (LDA) on the encoded documents to obtain K topics, wherein K is a positive integer, and each topic of the K topics corresponds to a list of key clusters with cluster IDs; and
for each topic, replace the cluster IDs with corresponding cluster labels.
16. The non-transitory CRM of claim 15 , wherein to generate the set of utterance embeddings based on the set of utterances, the set of instructions is further configured to cause the processor to:
generate the set of utterance embeddings based on the set of utterances with a neural network sentence encoder comprising one of the following models:
a sentence bidirectional encoder representation from transformers (SBERT) sentence encoder,
a universal sentence encoder (USE), and
a multilingual universal sentence encoder (MUSE).
17. The non-transitory CRM of claim 15 , wherein to obtain the plurality of clusters, the set of instructions is further configured to cause the processor to:
clusterize, based on a first hyperparameter and a second hyperparameter, the set of utterance embeddings to obtain the plurality of clusters, the first hyperparameter indicating a minimum number of utterance embeddings in any cluster, the second hyperparameter indicating a maximum distance between any neighboring pair of utterance embeddings in any cluster.
18. The non-transitory CRM of claim 15 , wherein to clusterize the set of utterance embeddings to obtain the plurality of clusters, the set of instructions is further configured to cause the processor to:
partition the set of utterance embeddings into a plurality of subsets of utterance embeddings;
for each subset, clusterize the subset of utterance embeddings to obtain a subset of clusters; and
combine all subsets of clusters to form the plurality of clusters.