Latent concept analysis method
A method of constructing a dataset for identifying a plurality of latent concepts in a Natural Language Processing model is provided. The method includes executing a clustering process on a first dataset, preparing a second dataset, defining a hierarchical concept tag-set from the second dataset, and annotating the hierarchical concept tag-set.
1 . A method of constructing a dataset for identifying a plurality of latent concepts in a Natural Language Processing model, comprising:
executing a clustering process on a first dataset, wherein the clustering process outputs a plurality of clusters;
preparing a second dataset, separate from the first dataset, wherein the second dataset is a portion of the first dataset which comprises less than a totality of the first dataset;
tokenizing the second dataset into a plurality of tokens and extracting contextualized representations for each of the plurality of tokens, wherein input tokens of the plurality of tokens which are split into at least one of sub-words and multiple tokens are mean pooled to create an embedding for each input token;
defining a hierarchical concept tag-set from the second dataset using at least one core language property, wherein the at least one core language property includes syntax; and
annotating the hierarchical concept tag-set comprising identifying a meaning of each of the plurality of clusters based on linguistic relation.
2 . The method of claim 1 , wherein the clustering process is an agglomerative hierarchical clustering method.
3 . The method of claim 1 , wherein a minimum variance criterion is used in the clustering process.
4 . The method of claim 1 , wherein the clustering process outputs a pre-defined number of clusters.
5 . The method of claim 1 , wherein the second dataset is randomly selected from the first dataset.
6 . The method of claim 1 , wherein preparing the second dataset comprises:
extracting a plurality of contextualized representations; and
tokenizing a plurality of sentences.
7 . The method of claim 1 , wherein a plurality of core language properties are used to define the hierarchical concept tag-set.
8 . The method of claim 1 , wherein annotating the hierarchical concept tag-set further comprises:
combining a portion of the plurality of clusters based on the meaning identified.
9 . A method of constructing a dataset for identifying a plurality of latent concepts in a Natural Language Processing model, comprising:
executing a clustering process on a first dataset, wherein the clustering process outputs a plurality of clusters;
preparing a second dataset, separate from the first dataset, wherein the second dataset is a portion of the first dataset which comprises less than a totality of the first dataset;
tokenizing the second dataset into a plurality of tokens and extracting contextualized representations for each of the plurality of tokens, wherein input tokens of the plurality of tokens which are split into at least one of sub-words and multiple tokens are mean pooled to create an embedding for each input token;
defining a hierarchical concept tag-set from the second dataset using at least one core language property, wherein the at least one core language property includes syntax;
annotating the hierarchical concept tag-set comprising identifying a meaning of each of the plurality of clusters based on linguistic relation, outputting an annotated dataset based on the annotated hierarchical concept tag-set; and
executing a logistic regression classifier on the annotated dataset.
10 . The method of claim 9 , wherein a threshold is used on the logistic regression classifier.
11 . The method of claim 9 , wherein the logistic regression classifier is trained on the first dataset, outputting a final dataset.
12 . The method of claim 9 , wherein the clustering process is an agglomerative hierarchical clustering method.
13 . The method of claim 9 , wherein a minimum variance criterion is used in the clustering process.
14 . The method of claim 9 , wherein the clustering process outputs a pre-defined number of clusters.
15 . The method of claim 9 , wherein the second dataset is randomly selected from the first dataset.
16 . The method of claim 9 , wherein preparing the second dataset comprises:
extracting a plurality of contextualized representations; and
tokenizing a plurality of sentences.
17 . The method of claim 9 , wherein a plurality of core language properties are used to define the hierarchical concept tag-set.
18 . The method of claim 9 , wherein annotating the hierarchical concept tag-set further comprises:
combining a portion of the plurality of clusters based on the meaning identified.