Techniques for aggregating insights of textual data using hierarchical clustering
A system and method for discovering and aggregating themes. The method includes applying a trained clustering model to a plurality of textual data, wherein the trained clustering model determines at least one cluster of textual data based on a meaning of the textual data, wherein textual data of the at least one cluster is a portion of the plurality of textual data; generating a name, using a trained naming model, for each of the at least one cluster, wherein the generated name indicates a theme that represents the meaning of the textual data of the at least one cluster; analyzing the at least one cluster to determine a distribution metric of the at least one cluster; and generating a notification based on the determined distribution metric and the respective at least one cluster.
1 . A method for discovering and aggregating themes, further comprising:
applying a trained clustering model to a plurality of textual data, wherein the trained clustering model determines at least one cluster of textual data based on a meaning of the textual data, wherein textual data of the at least one cluster is a portion of the plurality of textual data;
generating a name, using a trained naming model, for each of the at least one cluster based on textual units in the respective at least one cluster, wherein the generated name indicates a theme that represents the meaning of the textual data of the at least one cluster;
analyzing the at least one cluster to determine a distribution metric of the at least one cluster; and
generating a notification, with respect to at least one theme, based on a plurality of rules applied to the determined distribution metric and the at least one cluster associated with the at least one theme.
2 . The method of claim 1 , further comprising:
segmenting the textual data of the plurality of textual data into textual units; and
rephrasing the textual units to generate normalized formats of the textual units.
3 . The method of claim 1 , wherein the trained clustering model is a hierarchical clustering model that determines at least one sub-cluster for each of the determined at least one cluster.
4 . The method of claim 1 , further comprising:
aggregating the determined distribution metric and at least one other distribution metric of the at least one cluster, wherein the at least one other distribution metric is previously determined for the respective at least one cluster; and
identifying a trend in the theme by comparing the aggregated determined distribution metric and the at least one other distribution metric.
5 . The method of claim 4 , further comprising:
triggering the generation of the notification upon determination that the identified change is beyond a predetermined threshold value.
6 . The method of claim 1 , wherein the distribution metric is a numerical value that indicates a quality of the at least one cluster, and wherein the distribution metric is at least one of: Silhouette scores, Dunn index, McClain-Rao index, and C-index.
7 . The method of claim 1 , wherein the textual data is collected from at least one of: videoconferences, telephonic conversations, emails, text messages, chats, and customer relationship management (CRM) data.
8 . The method of claim 1 , wherein the textual data of the at least one cluster is related to at least one of: sales and customer relationships.
9 . The method of claim 1 , further comprising:
receiving an external input data from an external source; and
filtering the plurality of textual data by mapping to the external input data, wherein a portion of the plurality of textual data that is not mapped is filtered out.
10 . The method of claim 1 , wherein the trained clustering model applies any one of: hierarchical Latent Dirichlet Allocation (LDA), Bidirectional Encoder Representations from Transformers leveraged topic modeling (BERTopic), or hierarchical Density-based spatial clustering of applications (DBSCAN).
11 . A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:
applying a trained clustering model to a plurality of textual data, wherein the trained clustering model determines at least one cluster of textual data based on a meaning of the textual data, wherein textual data of the at least one cluster is a portion of the plurality of textual data;
generating a name, using a trained naming model, for each of the at least one cluster based on textual units in the respective at least one cluster, wherein the generated name indicates a theme that represents the meaning of the textual data of the at least one cluster;
analyzing the at least one cluster to determine a distribution metric of the at least one cluster; and
generating a notification, with respect to at least one theme, based on a plurality of rules applied to the determined distribution metric and the at least one cluster associated with the at least one theme.
12 . A system for discovering and aggregating themes, comprising:
a processing circuitry; and
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
apply a trained clustering model to a plurality of textual data, wherein the trained clustering model determines at least one cluster of textual data based on a meaning of the textual data, wherein textual data of the at least one cluster is a portion of the plurality of textual data;
generate a name, using a trained naming model, for each of the at least one cluster based on textual units in the respective at least one cluster, wherein the generated name indicates a theme that represents the meaning of the textual data of the at least one cluster;
analyze the at least one cluster to determine a distribution metric of the at least one cluster; and
generate a notification, with respect to at least one theme, based on a plurality of rules applied to the determined distribution metric and the at least one cluster associated with the at least one theme.
13 . The system of claim 12 , wherein the system is further configured to:
segment the textual data of the plurality of textual data into textual units; and
rephrase the textual units to generate normalized formats of the textual units.
14 . The system of claim 12 , wherein the trained clustering model is a hierarchical clustering model that determines at least one sub-cluster for each of the determined at least one cluster.
15 . The system of claim 12 , wherein the system is further configured to:
aggregate the determined distribution metric and at least one other distribution metric of the at least one cluster, wherein the at least one other distribution metric is previously determined for the respective at least one cluster; and
identify a trend in the theme by comparing the aggregated determined distribution metric and the at least one other distribution metric.
16 . The system of claim 15 , wherein the system is further configured to:
trigger the generation of the notification upon determination that the identified change is beyond a predetermined threshold value.
17 . The system of claim 12 , wherein the distribution metric is a numerical value that indicates a quality of the at least one cluster, and wherein the distribution metric is at least one of: Silhouette scores, Dunn index, McClain-Rao index, and C-index.
18 . The system of claim 12 , wherein the textual data is collected from at least one of: videoconferences, telephonic conversations, emails, text messages, chats, and customer relationship management (CRM) data.
19 . The system of claim 12 , wherein the textual data of the at least one cluster is related to at least one of: sales and customer relationships.
20 . The system of claim 12 , wherein the system is further configured to:
receive an external input data from an external source; and
filter the plurality of textual data by mapping to the external input data, wherein a portion of the plurality of textual data that is not mapped is filtered out.
21 . The system of claim 12 , wherein the trained clustering model applies any one of: hierarchical Latent Dirichlet Allocation (LDA), Bidirectional Encoder Representations from Transformers leveraged topic modeling (BERTopic), or hierarchical Density-based spatial clustering of applications (DBSCAN).