Document classification using unsupervised text analysis with concept extraction
View Patent ↗An embodiment for classifying documents using unsupervised text analysis with concept extraction. The embodiment may obtain a generic description for a document by: extracting concepts from headings and descriptions of a target document using natural language processing, matching the extracted concepts to generic abstracts obtained from an abstract database, where the abstract database is independent and separate from the target document, and processing the generic abstracts to lemmatize words and remove common words to form the generic description. The embodiment may process an available series of technical classifications. The embodiment may perform bidirectional analysis to determine a most relevant technical classification for the target document based on the generic description for the target document. The embodiment may assign the most relevant technical classification to the target document.
1 . A computer-based method for document classification, the method comprising:
obtaining a generic description for a target document, wherein obtaining the generic description comprises:
extracting concepts from headings and descriptions of the target document using natural language processing;
matching the extracted concepts to generic abstracts obtained from an abstract database, where the abstract database is independent and separate from the target document;
processing the generic abstracts to lemmatize words and remove common words to form the generic description;
processing an available series of technical classifications;
performing bidirectional analysis to determine a most relevant technical classification for the target document based on the generic description for the target document by summing occurrences of corresponding words within the generic description for the target document and aggregate descriptions of each of the technical classifications in the available series of technical classifications and calculating scores for each of the technical classifications in the available series of technical classifications based on the sum of occurrences; and
assigning the most relevant technical classification to the target document, wherein the most relevant technical classification corresponds to the technical classification with a highest calculated score, wherein the highest calculated score corresponds to a highest volume of word matches and a highest association between level descriptions of the technical classifications in the available series of technical classifications as compared to the generic description of the target document.
2 . The computer-based method of claim 1 , further comprising:
generating a word frequency table associated with the obtained generic description of the target document.
3 . The computer-based method of claim 1 , further comprising:
concatenating the level descriptions within a tree structure corresponding to each of the technical classifications in the available series of technical classifications;
further processing each of the technical classifications in the available series of technical classifications by lemmatizing the words and removing common non-technical words to obtain aggregate descriptions; and
generating word tables for each of the aggregate descriptions of each of the technical classifications in the available series of technical classifications.
4 . The computer-based method of claim 1 , wherein the extracted concepts from headings and descriptions of the target document are obtained using multiple large language models, and wherein at least a portion of the extracted concepts are abstract and non-technical in substance.
5 . The computer-based method of claim 1 , wherein the available series of technical classifications is derived from an available set of generic technical classifications.
6 . A computer system, the computer system comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more computer-readable tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
obtaining a generic description for a target document, wherein obtaining the generic description comprises:
extracting concepts from headings and descriptions of the target document using natural language processing;
matching the extracted concepts to generic abstracts obtained from an abstract database, where the abstract database is independent and separate from the target document;
processing the generic abstracts to lemmatize words and remove common words to form the generic description;
processing an available series of technical classifications;
performing bidirectional analysis to determine a most relevant technical classification for the target document based on the generic description for the target document by summing occurrences of corresponding words within the generic description for the target document and aggregate descriptions of each of the technical classifications in the available series of technical classifications and calculating scores for each of the technical classifications in the available series of technical classifications based on the sum of occurrences; and
assigning the most relevant technical classification to the target document, wherein the most relevant technical classification corresponds to the technical classification with a highest calculated score, wherein the highest calculated score corresponds to a highest volume of word matches and a highest association between level descriptions of the technical classifications in the available series of technical classifications as compared to the generic description of the target document.
7 . The computer system of claim 6 , further comprising:
generating a word frequency table associated with the obtained generic description of the target document.
8 . The computer system of claim 6 , further comprising:
concatenating the level descriptions within a tree structure corresponding to each of the technical classifications in the available series of technical classifications;
further processing each of the technical classifications in the available series of technical classifications by lemmatizing the words and removing common non-technical words to obtain aggregate descriptions; and
generating word frequency tables for each of the aggregate descriptions of each of the technical classifications in the available series of technical classifications.
9 . The computer system of claim 6 , wherein the extracted concepts from headings and descriptions of the target document are obtained using multiple large language models, and wherein at least a portion of the extracted concepts are abstract and non-technical in substance.
10 . The computer system of claim 6 , wherein the available series of technical classifications is derived from an available set of generic technical classifications.
11 . A computer program product comprising:
one or more computer-readable storage media;
program instructions stored on the one or more computer-readable storage media to perform operations comprising:
obtaining a generic description for a target document, wherein obtaining the generic description comprises:
extracting concepts from headings and descriptions of the target document using natural language processing;
matching the extracted concepts to generic abstracts obtained from an abstract database, where the abstract database is independent and separate from the target document;
processing the generic abstracts to lemmatize words and remove common words to form the generic description;
processing an available series of technical classifications;
performing bidirectional analysis to determine a most relevant technical classification for the target document based on the generic description for the target document by summing occurrences of corresponding words within the generic description for the target document and aggregate descriptions of each of the technical classifications in the available series of technical classifications and calculating scores for each of the technical classifications in the available series of technical classifications based on the sum of occurrences; and
assigning the most relevant technical classification to the target document, wherein the most relevant technical classification corresponds to the technical classification with a highest calculated score, wherein the highest calculated score corresponds to a highest volume of word matches and a highest association between level descriptions of the technical classifications in the available series of technical classifications as compared to the generic description of the target document.
12 . The computer program product of claim 11 , further comprising:
generating a word frequency table associated with the generic description of the target document.
13 . The computer program product of claim 11 , further comprising:
concatenating the level descriptions within a tree structure corresponding to each of the technical classifications in the available series of technical classifications;
further processing each of the technical classifications in the available series of technical classifications by lemmatizing the words and removing common non-technical words to obtain aggregate descriptions; and
generating word frequency tables for each of the aggregate descriptions of each of the technical classifications in the available series of technical classifications.
14 . The computer program product of claim 11 , wherein the extracted concepts from headings and descriptions of the target document are obtained using multiple large language models, and wherein at least a portion of the extracted concepts are abstract and non-technical in substance.
15 . The computer program product of claim 14 , wherein the available series of technical classifications is derived from an available set of generic technical classifications.
16 . The computer-based method of claim 5 , wherein vector representations of the technical classifications are leveraged to extract the concepts.
17 . The computer-based method of claim 16 , wherein the extracted concepts are utilized in obtaining independent abstracts.
18 . The computer system of claim 10 , wherein vector representations of the technical classifications are leveraged to extract the concepts.
19 . The computer system of claim 18 , wherein the extracted concepts are utilized in obtaining independent abstracts.
20 . The computer program product of claim 15 , wherein vector representations of the technical classifications are leveraged to extract the concepts, and wherein the extracted concepts are utilized in obtaining independent abstracts.