Method and system for document similarity analysis

1. A method for document similarity analysis, the method comprising:

receiving, by a processor, a selection of a reference document specified by a user via a user interface;

converting, by the processor, the reference document into a reference document content identifier from content of the reference document, the converting further comprising:

processing the content of the reference document to identify a plurality of frequently occurring terms contained in the content of the reference document;

encoding the plurality of frequently occurring terms to generate encoded frequently occurring terms, encoding the frequently occurring terms comprising mapping the plurality of frequently occurring terms to respective numbers; and

generating the reference document content identifier, wherein generating the reference document content identifier comprises combining the encoded frequently occurring terms into a bit array representation populated with the encoded frequently occurring terms, the bit array representation having bits set at locations, each bit representing a particular one of the frequently occurring terms from the plurality of frequently occurring terms;

comparing the reference document content identifier with a plurality of archived document content identifiers, each associated with one of a plurality of archived documents stored in a document repository, the comparing including determining a degree of difference between reference document content and archived document content of each of the plurality of archived documents based on a count of deviating bits between the reference document content identifier and each of the plurality of archived document content identifiers;

generating a plurality of document similarity values based on the determined degree of difference between reference document content and archived document content of each of the plurality of archived documents determined by the comparing of the reference document content identifier with the plurality of archived document content identifiers; and

returning, by the processor, a document list for presentation on the user interface, the document list listing a set of documents identified from the plurality of archived documents based on the plurality of document similarity values.

2. The method of claim 1 , further comprising generating a binary search tree representing the plurality of archived document content identifiers, and wherein:

the binary search tree comprises a plurality of nodes, the plurality of nodes comprising:

branch nodes; and

key nodes, wherein each key node represents a portion of a corresponding archived document identifier from the plurality of archived document content identifiers where no further branching occurs;

generating the plurality of document similarity values based on the determined degree of difference between reference document content and archived document content of each of the plurality of archived documents determined by the comparing of the reference document content identifier with the plurality of archived document content identifiers comprises:

traversing a branch having a lowest number of deviating bits from the reference document content identifier to a terminating key node; and

determining a total number of deviating bits.

3. The method of claim 1 , wherein encoding the plurality of frequently occurring terms comprises applying a hash function to each the plurality of frequently occurring terms to generate a plurality of hashed values.

4. The method of claim 3 , wherein encoding the plurality of frequently occurring terms further comprises generating a plurality of term identifiers, the plurality of term identifiers comprising a term identifier for each of the plurality of frequently occurring terms, wherein generating the plurality of term identifiers comprises applying a modulo operation on each of the plurality of hashed values.

5. The method of claim 1 , wherein encoding the plurality of frequently occurring terms includes applying an algorithm to each of the plurality of frequently occurring terms to generate outputs having a specified length.

6. The method of claim 1 , wherein converting the reference document into the reference document content identifier further comprises, prior to identifying the plurality of frequently occurring terms in the content of the reference document:

removing stop words from the content of the reference document; and

stemming the content of the reference document.

7. The method of claim 1 , wherein the plurality of archived document content identifiers are organized as binary search trees.

8. The method of claim 7 , wherein generating the plurality of document similarity values further comprises traversing the binary search trees to identify an archived document content identifier with a highest similarity to the reference document content identifier.

9. A computer program product comprising a non-transitory, computer-readable medium storing thereon computer-executable instructions executable by a processor, the computer-executable instructions comprising instructions for:

receiving a selection of a reference document specified by a user via a user interface;

converting the reference document into a reference document content identifier from content of the reference document, the converting further comprising:

processing the content of the reference document to identify a plurality of frequently occurring terms contained in the content of the reference document;

encoding the plurality of frequently occurring terms to generate encoded frequently occurring terms, encoding the frequently occurring terms comprising mapping the plurality of frequently occurring terms to respective numbers; and

generating the reference document content identifier, wherein generating the reference document content identifier comprises combining the encoded frequently occurring terms into a bit array representation populated with the encoded frequently occurring terms, the bit array representation having bits set at locations, each bit representing a particular one of the frequently occurring terms from the plurality of frequently occurring terms;

comparing the reference document content identifier with a plurality of archived document content identifiers, each associated with one of a plurality of archived documents stored in a document repository, the comparing including determining a degree of difference between reference document content and archived document content of each of the plurality of archived documents based on a count of deviating bits between the reference document content identifier and each of the plurality of archived document content identifiers;

generating a plurality of document similarity values based on the determined degree of difference between reference document content and archived document content of each of the plurality of archived documents determined by the comparing of the reference document content identifier with the plurality of archived document content identifiers; and

returning a document list for presentation on the user interface, the document list listing a set of documents identified from the plurality of archived documents based on the plurality of document similarity values.

10. The computer program product of claim 9 , wherein the computer-executable instructions further comprise instructions for generating a binary search tree representing the plurality of archived document content identifiers, and wherein:

the binary search tree comprises a plurality of nodes, the plurality of nodes comprising:

branch nodes; and

key nodes, wherein each key node represents a portion of a corresponding archived document identifier from the plurality of archived document content identifiers where no further branching occurs;

generating the plurality of document similarity values based on the determined degree of difference between reference document content and archived document content of each of the plurality of archived documents determined by the comparing of the reference document content identifier with the plurality of archived document content identifiers comprises:

traversing a branch having a lowest number of deviating bits from the reference document content identifier to a terminating key node; and

determining a total number of deviating bits.

11. The computer program product of claim 9 , wherein encoding the plurality of frequently occurring terms comprises applying a hash function to each the plurality of frequently occurring terms to generate a plurality of hashed values.

12. The computer program product of claim 11 , wherein encoding the plurality of frequently occurring terms further comprises generating a plurality of term identifiers, the plurality of term identifiers comprising a term identifier for each of the plurality of frequently occurring terms, wherein generating the plurality of term identifiers comprises applying a modulo operation on each of the plurality of hashed values.

13. The computer program product of claim 9 , wherein encoding the plurality of frequently occurring terms includes applying an algorithm to each of the plurality of frequently occurring terms to generate outputs having a specified length.

14. The computer program product of claim 9 , wherein converting the reference document into the reference document content identifier further comprises, prior to identifying the plurality of frequently occurring terms in the content of the reference document:

removing stop words from the content of the reference document; and

stemming the content of the reference document.

15. The computer program product of claim 9 , wherein the plurality of archived document content identifiers are organized as binary search trees.

16. The computer program product of claim 15 , wherein generating the plurality of document similarity values further comprises traversing the binary search trees to identify an archived document content identifier with a highest similarity to the reference document content identifier.

17. A system for document similarity analysis comprising:

a document repository comprising:

a plurality of archived documents; and

a plurality of archived document content identifiers, each associated with one of the plurality of archived documents;

a user interface;

a processor; and

a memory storing computer-executable instructions executable by the processor, the computer-executable instructions comprising instructions for:

receiving a selection of a reference document specified by a user via the user interface;

converting the reference document into a reference document content identifier from content of the reference document, the converting further comprising:

processing the content of the reference document to identify a plurality of frequently occurring terms contained in the content of the reference document;

encoding the plurality of frequently occurring terms to generate encoded frequently occurring terms, encoding the frequently occurring terms comprising mapping the plurality of frequently occurring terms to respective numbers; and

generating the reference document content identifier, wherein generating the reference document content identifier comprises combining the encoded frequently occurring terms into a bit array representation populated with the encoded frequently occurring terms, the bit array representation having bits set at locations, each bit representing a particular one of the frequently occurring terms from the plurality of frequently occurring terms;

comparing the reference document content identifier with the plurality of archived document content identifiers, the comparing including determining a degree of difference between reference document content and archived document content of each of the plurality of archived documents based on a count of deviating bits between the reference document content identifier and each of the plurality of archived document content identifiers;

generating a plurality of document similarity values based on the determined degree of difference between reference document content and archived document content of each of the plurality of archived documents determined by the comparing of the reference document content identifier with the plurality of archived document content identifiers; and

returning a document list for presentation on the user interface, the document list listing a set of documents identified from the plurality of archived documents based on the plurality of document similarity values.

18. The system of claim 17 , wherein the computer-executable instructions further comprise instructions for generating a binary search tree representing the plurality of archived document content identifiers, and wherein:

the binary search tree comprises a plurality of nodes, the plurality of nodes comprising:

branch nodes; and

key nodes, wherein each key node represents a portion of a corresponding archived document identifier from the plurality of archived document content identifiers where no further branching occurs;

generating the plurality of document similarity values based on the determined degree of difference between reference document content and archived document content of each of the plurality of archived documents determined by the comparing of the reference document content identifier with the plurality of archived document content identifiers comprises:

traversing a branch having a lowest number of deviating bits from the reference document content identifier to a terminating key node; and

determining a total number of deviating bits.

19. The system of claim 17 , wherein encoding the plurality of frequently occurring terms comprises applying a hash function to each the plurality of frequently occurring terms to generate a plurality of hashed values.

20. The system of claim 19 , wherein encoding the plurality of frequently occurring terms further comprises generating a plurality of term identifiers, the plurality of term identifiers comprising a term identifier for each of the plurality of frequently occurring terms, wherein generating the plurality of term identifiers comprises applying a modulo operation on each of the plurality of hashed values.

21. The system of claim 17 , wherein converting the reference document into the reference document content identifier further comprises, prior to identifying the plurality of frequently occurring terms in the content of the reference document:

removing stop words from the content of the reference document; and

stemming the content of the reference document.

22. The system of claim 17 , wherein the plurality of archived document content identifiers are organized as binary search trees.