Theme detection within a corpus of information
Systems and methods are used to detect underlying themes from a collection of documents at an aggregated level. A representative set of documents may be selected from a cluster of documents, with the representative set of documents corresponding to a general theme of the cluster. Candidate theme phrases may then be extracted from the documents and used to generate document embeddings and candidate phrase embeddings, which may be ranked, such as with a diversity-based ranking approach. Certain candidates may be selected from the ranking. Each of the documents forming the representative set may then be concatenated and a query embedding may be generated and ranked against the candidate phrases. In this manner, a collection of phrases associated with both the general underlying theme of the cluster, along with granular topics associated with that theme, may be identified.
1. A computer-implemented method, comprising:
retrieving a set of documents including unstructured text associated with a cluster, the cluster grouping the set of documents based on one or more central themes;
selecting, from the set of documents, a representative set;
extracting, from portions of the unstructured text in respective documents forming the representative set, one or more candidate theme phrases;
generating, for each document forming the representative set, a document embedding corresponding to individual candidate theme phrases extracted from the document;
generating a candidate phrase embedding corresponding to a set of candidate theme phrases collected from each document of the representative set;
ranking the one or more candidate theme phrases based, at least in part, on the respective document embeddings and the candidate phrase embedding;
selecting, from the ranking, a set of retained candidate theme phrases;
generating, from the representative set, a query embedding corresponding to a concatenated combination of the representative set;
ranking the one or more candidate theme phrases from the query embedding and the set of retained candidate theme phrases;
selecting, based on the ranking, a refined set of retained candidate theme phrases corresponding to a set of semantically similar retained candidate theme phrases associated with the representative set;
determining a retained candidate theme phrase of the refined set of retained candidate theme phrases corresponds to a workflow item; and
changing a priority level for the workflow item.
2. The computer-implemented method of claim 1 , further comprising:
generating a query document corresponding to a concatenated collection of documents in the representative set.
3. The computer-implemented method of claim 1 , wherein at least one of the rankings is based on maximal marginal relevance.
4. The computer-implemented method of claim 1 , further comprising:
mapping the refined set of candidate theme phrases back to respective originating phrases.
5. A method comprising:
determining a representative set of documents from a cluster, based at least in part on one or more metrics associated with a target number of documents;
generating, for each document in the representative set, one or more document embeddings for extracted candidate theme phrases in the form of one or more bi-grams or tri-grams;
generating, for extracted theme phrases from the representative set, one or more embeddings;
identifying, using a diversity-based ranking approach, a set of theme phrases from the one or more document embeddings and the one or more embeddings;
generating, for the representative set of documents, a query embedding including information from each document of the representative set of documents; and
selecting, using the diversity-based ranking approach, a ranked set of theme phrases having a semantic similarity to at least a portion of the cluster and being diverse from other theme phrases in the ranked set of theme phrases, based at least on the query embedding and the set of theme phrases.
6. The method of claim 5 , wherein the query embedding is based on a concatenated collection of each document in the representative set of documents.
7. The method of claim 5 , wherein the diversity-based ranking approach is maximum marginal relevance.
8. The method of claim 5 , wherein the cluster is generated using a centroid-based clustering algorithm.
9. The method of claim 5 , further comprising:
extracting, from the documents forming the representative set, one or more candidate theme phrases; and
removing one or more portions of the candidate theme phrases to generate the theme phrases.
10. The method of claim 9 , wherein the removing includes removing stop words.
11. The method of claim 5 , wherein the extracted theme phrases include non-contiguous tokens.
12. The method of claim 5 , further comprising:
mapping the ranked set of theme phrases back to a respective originating phrase.
13. The method of claim 5 , wherein the theme phrases are extracted using a sentence transformer encoder.
14. The method of claim 5 , further comprising:
generating an action item based on a ranked theme phrase of the ranked set of theme phrases; and
providing a notification to a user including the action item.
15. The method of claim 5 , wherein the representative set includes a number of documents based on a respective distance from a center of the cluster.
16. A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
determine a representative set of documents from a cluster, based at least in part on one or more metrics associated with a target number of documents;
generate, for each document in the representative set, one or more document embeddings for extracted candidate theme phrases in the form of one or more bi-grams or tri-grams;
generate, for extracted theme phrases from the representative set, one or more embeddings;
identify, using a diversity-based ranking approach, a set of theme phrases from the one or more document embeddings and the one or more embeddings;
generate, for the representative set of documents, a query embedding including information from each document of the representative set of documents; and
select, using the diversity-based ranking approach, a ranked set of theme phrases having a semantic similarity to at least a portion of the cluster and being diverse from other theme phrases in the ranked set of theme phrases, based at least on the query embedding and the set of theme phrases.
17. The system of claim 16 , wherein the query embedding is based on a concatenated collection of each document in the representative set of documents.
18. The system of claim 16 , wherein the diversity-based ranking approach is maximum marginal relevance.
19. The system of claim 16 , wherein the cluster is generated using a centroid-based clustering algorithm.
20. The system of claim 16 , wherein the instructions when executed further cause the system to:
map the ranked set of theme phrases back to a respective originating phrase.