Method and system for document similarity analysis based on weak transitive relation of similarity
A method for document similarity analysis. The method includes obtaining a document to be archived, obtaining indexing terms for the document to be archived, and identifying document categories similar to the document to be archived, based on indexing terms and corresponding term frequencies. The method further includes making a determination that a weakly similar document category exists and based on the determination: generating a new document category and registering the document in the new document category, and registering the document in the weakly similar document category.
1. A method for document similarity analysis, the method comprising:
obtaining, by a document categorization and search engine executing on a processor, a first document;
obtaining, for the first document, first document index terms;
identifying a first document category similar to the first document, based on first category index term frequencies;
calculating a first similarity value between the first document and the first document category;
making a first determination that the first similarity value does not exceed a strong similarity threshold;
making a second determination that the first similarity value exceeds a weak similarity threshold; and
based on the first and second determinations:
making a first association between the first document and the first document category, wherein the first association specifies a weak similarity;
generating a second document category; and
making a second association between the first document and the second document category.
2. The method of claim 1 , wherein the first similarity value is calculated using a cosine similarity between first document index term frequencies and the first category index term frequencies.
3. The method of claim 2 , wherein the first similarity value exceeds the weak similarity threshold when the first similarity value is at least 0.2.
4. The method of claim 1 , wherein generating the second document category comprises:
assigning the first document index terms and first document index term frequencies to the second document category.
5. The method of claim 1 , further comprising:
obtaining a second document;
obtaining, for the second document, second document index terms and second document index term frequencies;
identifying a third document category similar to the second document, based on the second document index terms and the second document index term frequencies;
calculating a second similarity value between the second document and the third document category;
making a third determination that the second similarity value exceeds the strong similarity threshold; and
based on the third determination:
making a third association between the second document and the third document category.
6. The method of claim 5 , wherein;
the second similarity value is a cosine similarity; and
the second similarity value exceeds the strong similarity threshold when the second similarity value is at least 0.78.
7. The method of claim 1 , further comprising:
obtaining a target document;
obtaining target index terms for the target document;
identifying a target document category similar to the target document, based on the target index terms; and
identifying, in the target document category, a similar document.
8. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, perform a method comprising:
obtaining a first document;
obtaining, for the first document, first document index terms;
identifying a first document category similar to the first document, based on first category index term frequencies;
calculating a first similarity value between the first document and the first document category;
making a first determination that the first similarity value does not exceed a strong similarity threshold;
making a second determination that the first similarity value exceeds a weak similarity threshold; and
based on the first and second determinations:
making a first association between the first document and the first document category, wherein the first association specifies a weak similarity;
generating a second document category; and
making a second association between the first document and the second document category.
9. The non-transitory computer-readable medium of claim 8 , wherein the first similarity value is calculated using a cosine similarity between first document index term frequencies and the first category index term frequencies.
10. The non-transitory computer-readable medium of claim 8 , wherein generating the second document category comprises:
assigning the first document index terms and first document index term frequencies to the second document category.
11. The non-transitory computer-readable medium of claim 8 , wherein the method further comprises:
obtaining a second document;
obtaining, for the second document, second document index terms and second document index term frequencies;
identifying a third document category similar to the second document, based on the second document index terms and the second document index term frequencies;
calculating a second similarity value between the second document and the third document category;
making a third determination that the second similarity value exceeds the strong similarity threshold; and
based on the third determination:
making a third association between the second document and the third document category.
12. The non-transitory computer-readable medium of claim 8 , wherein the method further comprises:
obtaining a target document;
obtaining target index terms for the target document;
identifying a target document category similar to the target document, based on the target index terms; and
identifying, in the target document category, a similar document.
13. A system for document similarity analysis comprising:
a document repository;
memory, wherein the memory comprises instructions; and
a processor, executing a document categorization and search engine, wherein the instructions are executed by the processor to perform a method comprising:
obtaining, by the document categorization and search engine, a first document;
obtaining, for the first document, first document index terms;
identifying, in the document repository, a first document category similar to the first document, based on first category index term frequencies;
calculating a first similarity value between the first document and the first document category;
making a first determination that the first similarity value does not exceed a strong similarity threshold;
making a second determination that the first similarity value exceeds a weak similarity threshold; and
based on the first and second determinations:
making a first association between the first document and the first document category, wherein the first association specifies a weak similarity;
generating a second document category; and
making a second association between the first document and the second document category.
14. The system of claim 13 , wherein the first similarity value is calculated using a cosine similarity between first document index term frequencies and the first category index term frequencies.
15. The system of claim 13 , wherein generating the second document category comprises:
assigning the first document index terms and first document index term frequencies to the second document category.
16. The system of claim 13 , wherein the method further comprises:
obtaining a second document;
obtaining, for the second document, second document index terms and second document index term frequencies;
identifying, in the document repository, a third document category similar to the second document, based on the second document index terms and the second document index term frequencies;
calculating a second similarity value between the second document and the third document category;
making a third determination that the second similarity value exceeds the strong similarity threshold; and
based on the third determination:
making a third association between the second document and the third document category.
17. The system of claim 13 , wherein the method further comprises:
obtaining a target document;
obtaining target index terms for the target document;
identifying, in the document repository, a target document category similar to the target document, based on the target index terms; and
identifying, in the target document category, a similar document.