Method and system for document similarity analysis

1. A method for document similarity analysis, the method comprising:

receiving, by a processor, a selection of a reference document specified by a user via a user interface;

converting, by the processor, the selected reference document into a reference document content identifier from content of the selected reference document, the converting further comprising:

tokenizing the content of the selected reference document;

identifying a plurality of frequently occurring terms in the tokenized content of the reference document;

for each of the identified frequently occurring terms, encoding the respective identified frequently occurring term by mapping the respective identified frequently occurring term to a number; and

generating the reference document content identifier based on the encoded plurality of frequently occurring terms by combining the encoded plurality of frequently occurring terms into a bit array representation populated with the encoded plurality of frequently occurring terms, the bit array representation having bits each representing a particular one of the identified frequently occurring terms and set at locations each representing a particular one of the identified frequently occurring terms and the identified frequently occurring terms is contained in the reference document;

comparing the reference document content identifier with a plurality of archived document content identifiers, each associated with one of a plurality of archived documents stored in a document repository;

generating a plurality of document similarity values based on comparisons of the reference document content identifier with the plurality of archived document content identifiers; and

returning, by the processor, a document list for presentation on the user interface, the document list listing a set of documents identified from the plurality of archived documents based on the plurality of document similarity values.

2. The method of claim 1 , wherein encoding the identified frequently occurring terms includes applying a hash function to each of the identified frequently occurring terms.

3. The method of claim 2 , wherein encoding the identified frequently occurring terms further comprises applying a modulo operation on each of the hashed values to generate a term identifier for each of the identified frequently occurring terms.

4. The method of claim 1 , wherein encoding the identified frequently occurring terms includes applying an algorithm to each of the identified frequently occurring terms to generate outputs having a specified length.

5. The method of claim 1 , wherein converting the selected reference document into a reference document content identifier further comprises, prior to identifying the frequently occurring terms in the content of the reference document:

removing stop words from the content of the reference document; and

stemming the content of the reference document.

6. The method of claim 1 , wherein the plurality of archived document content identifiers are each organized in a binary search tree.

7. The method of claim 6 , wherein generating the plurality of document similarity values further comprises traversing the respective binary search trees to identify an archived document content identifier with a highest similarity to the reference document content identifier.

8. A system for document similarity analysis, comprising:

a computing device comprising a computer processor;

a document content identifier encoding engine, executable on the computer processor to:

receive, by a processor, a selection of a reference document specified by a user via a user interface;

convert, by the processor, the selected reference document into a reference document content identifier from content of the selected reference document, the converting further comprising:

tokenize the content of the selected reference document;

identify a plurality of frequently occurring terms in the tokenized content of the reference document;

for each of the identified frequently occurring terms, encode the respective identified frequently occurring term by mapping the respective identified frequently occurring term to a number; and

generate the reference document content identifier based on the encoded plurality of frequently occurring terms by combining the encoded plurality of frequently occurring terms into a bit array representation populated with the encoded plurality of frequently occurring terms, the bit array representation having bits each representing a particular one of the identified frequently occurring terms and set at locations each representing a particular one of the identified frequently occurring terms and the identified frequently occurring terms is contained in the reference document;

compare the reference document content identifier with a plurality of archived document content identifiers, each associated with one of a plurality of archived documents stored in a document repository;

generate a plurality of document similarity values based on comparisons of the reference document content identifier with the plurality of archived document content identifiers; and

return, by the processor, a document list for presentation on the user interface, the document list listing a set of documents identified from the plurality of archived documents based on the plurality of document similarity values.

9. The system of claim 8 , wherein encoding the identified frequently occurring terms includes applying a hash function to each of the identified frequently occurring terms.

10. The system of claim 9 , wherein encoding the identified frequently occurring terms further comprises applying a modulo operation on each of the hashed values to generate a term identifier for each of the identified frequently occurring terms.

11. The system of claim 8 , wherein encoding the identified frequently occurring terms includes applying an algorithm to each of the identified frequently occurring terms to generate outputs having a specified length.

12. The system of claim 8 , wherein converting the selected reference document into a reference document content identifier further comprises, prior to identifying the frequently occurring terms in the content of the reference document:

removing stop words from the content of the reference document; and

stemming the content of the reference document.

13. The system of claim 8 , wherein the plurality of archived document content identifiers is each organized in a binary search tree.

14. The system of claim 13 , wherein generating the plurality of document similarity values further comprises traversing the respective binary search trees to identify an archived document content identifier with a highest similarity to the reference document content identifier.

15. A non-transitory computer readable medium storing computer-readable instructions for document similarity analysis, the computer-readable instructions executable to:

receive, by a processor, a selection of a reference document specified by a user via a user interface;

convert, by the processor, the selected reference document into a reference document content identifier from content of the selected reference document, the converting further comprising:

tokenize the content of the selected reference document;

identify a plurality of frequently occurring terms in the tokenized content of the reference document;

for each of the identified frequently occurring terms, encode the respective identified frequently occurring term by mapping the respective identified frequently occurring term to a number; and

generate the reference document content identifier based on the encoded plurality of frequently occurring terms by combining the encoded plurality of frequently occurring terms into a bit array representation populated with the encoded plurality of frequently occurring terms, the bit array representation having bits each representing a particular one of the identified frequently occurring terms and set at locations each representing a particular one of the identified frequently occurring terms and the identified frequently occurring terms is contained in the reference document;

compare the reference document content identifier with a plurality of archived document content identifiers, each associated with one of a plurality of archived documents stored in a document repository;

generate a plurality of document similarity values based on comparisons of the reference document content identifier with the plurality of archived document content identifiers; and

return, by the processor, a document list for presentation on the user interface, the document list listing a set of documents identified from the plurality of archived documents based on the plurality of document similarity values.

16. The non-transitory computer readable medium of claim 15 , wherein encoding the identified frequently occurring terms includes applying a hash function to each of the identified frequently occurring terms.

17. The non-transitory computer readable medium of claim 15 , wherein converting the selected reference document into a reference document content identifier further comprises, prior to identifying the frequently occurring terms in the content of the reference document:

removing stop words from the content of the reference document; and

stemming the content of the reference document.

18. The non-transitory computer readable medium of claim 15 , wherein the plurality of archived document content identifiers is each organized in a binary search tree.

19. The non-transitory computer readable medium of claim 18 , wherein generating the plurality of document similarity values further comprises traversing the respective binary search trees to identify an archived document content identifier with a highest similarity to the reference document content identifier.