Identifying similar documents in a file repository using unique document signatures
Methods, systems, and non-transitory computer readable storage media are disclosed for determining clusters of similar digital documents using unique document signatures. Specifically, the disclosed system processes digital text in a digital document to tokenize character strings (e.g., words) in the digital document by combining a subset of character values and string lengths in the character strings. Additionally, the disclosed system generates a document signature for the digital document by combining subsets of tokens generated for the digital document into a token sequence indicative of the digital text in the digital document. The disclosed system determines a cluster of similar digital documents including the digital document by comparing the document signature of the digital document to document signatures corresponding to a plurality of digital documents.
1. A computer-implemented method comprising:
generating, by at least one processor utilizing digital text processing, a plurality of tokens representing a plurality of character strings from a digital document of a plurality of digital documents stored at a digital content database by:
determining, for a character string of the plurality of character strings, a plurality of character values of the character string;
determining a string length of the character string; and
generating, for the character string, a token comprising the plurality of character values and the string length of the character string;
generating, by the at least one processor, a document signature for the digital document by combining one or more subsets of tokens of the plurality of tokens into a token sequence; and
determining, by the at least one processor accessing the plurality of digital documents stored at the digital content database, a cluster of similar digital documents comprising the digital document by comparing the document signature to a plurality of document signatures corresponding to the plurality of digital documents.
2. The computer-implemented method of claim 1 , wherein determining the plurality of character values comprises:
determining a set of characters in a beginning portion of the character string.
3. The computer-implemented method of claim 2 , wherein generating the token for the character string comprises appending the string length to the set of characters.
4. The computer-implemented method of claim 3 , wherein generating the token for the character string further comprises:
determining an additional set of characters in an ending portion of the character string; and
appending the additional set of characters after the string length.
5. The computer-implemented method of claim 4 , wherein:
determining the set of characters in the beginning portion comprises determining first two characters of the character string; and
determining the additional set of characters in the ending portion comprises determining last two characters of the character string.
6. The computer-implemented method of claim 1 , wherein generating the document signature comprises:
arranging the plurality of tokens in an order according to a frequency of occurrence of the plurality of character strings in the digital document; and
selecting the one or more subsets of tokens based on the order of the plurality of tokens.
7. The computer-implemented method of claim 6 , wherein generating the document signature further comprises:
determining a subset of tokens corresponding to character strings having a first subset of frequencies of occurrence in the digital document above a frequency threshold; and
generating the token sequence by concatenating the subset of tokens into the token sequence.
8. The computer-implemented method of claim 7 , wherein generating the document signature comprises:
determining an additional subset of tokens corresponding to character strings having a second subset of frequencies of occurrence in the digital document below the frequency threshold; and
generating the token sequence by concatenating the additional subset of tokens with the subset of tokens into the token sequence.
9. The computer-implemented method of claim 1 , wherein generating the document signature comprises:
determining a word count and a character count for the digital document based on the plurality of character strings; and
generating the document signature by combining the one or more subsets of tokens with the word count and the character count.
10. The computer-implemented method of claim 1 , wherein determining the cluster of similar digital documents comprises:
generating a linked list comprising the plurality of digital documents based on the plurality of document signatures corresponding to the plurality of digital documents; and
determining the cluster of similar digital documents comprising the digital document based on the linked list.
11. The computer-implemented method of claim 10 , wherein determining the cluster of similar digital documents comprises:
determining a similarity between a first document signature of a first digital document and a second document signature of a second digital document by comparing token values in the first document signature to token values in the second document signature; and
determining that the first digital document and the second digital document are similar in response to determining that the similarity between the first document signature and the second document signature meet a similarity threshold.
12. A system comprising:
one or more processors configured to cause the system to:
determine, utilizing digital text processing, character values and string lengths of a plurality of character strings in a digital document of a plurality of digital documents stored at a digital content database by determining the string lengths representing numbers of characters in corresponding character strings of the digital document;
generate a plurality of tokens representing the plurality of character strings by combining the character values and the string lengths of the plurality of character strings;
generate a document signature for the digital document by combining one or more subsets of tokens of the plurality of tokens into a token sequence; and
determine, by accessing the plurality of digital documents stored at the digital content database, a cluster of similar digital documents comprising the digital document by comparing the document signature to a plurality of document signatures corresponding to the plurality of digital documents.
13. The system of claim 12 , wherein the one or more processors are further configured to cause the system to determine the character values and the string lengths of the plurality of character strings by:
determining a first set of character values in a beginning portion of a character string of the plurality of character strings;
determining a second set of character values in an ending portion of the character string; and
determining a string length of the character string based on a number of characters in the character string.
14. The system of claim 13 , wherein the one or more processors are further configured to cause the system to generate, for the character string, a token by concatenating the first set of character values, the string length, and the second set of character values.
15. The system of claim 12 , wherein the one or more processors are further configured to cause the system to generate the document signature by:
arranging the plurality of tokens according to frequency of occurrence of corresponding character strings within the digital document;
determining the one or more subsets of tokens by selecting a predetermined number of tokens of the plurality of tokens based on the frequency of occurrence; and
concatenating the predetermined number of tokens into the token sequence.
16. The system of claim 15 , wherein the one or more processors are further configured to cause the system to:
select the predetermined number of tokens by:
determining a first subset of tokens comprising highest numbers of instances in the digital document; and
determining a second subset of tokens comprising lowest numbers of instances in the digital document; and
concatenate the predetermined number of tokens by concatenating the first subset of tokens and the second subset of tokens into the token sequence.
17. The system of claim 12 , wherein the one or more processors are further configured to cause the system to:
generate a hash table comprising the plurality of tokens representing the plurality of character strings based on the plurality of document signatures;
generate a linked list comprising the plurality of digital documents within the hash table based on the plurality of document signatures and a plurality of document locations of the plurality of digital documents; and
determine the cluster of similar digital documents comprising the digital document by comparing one or more token values of the plurality of document signatures.
18. A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to perform operations comprising:
generating, utilizing digital text processing, a plurality of tokens representing a plurality of character strings from a digital document of a plurality of digital documents stored at a digital content database by:
determining, for a character string of the plurality of character strings, a plurality of character values of the character string;
determining a string length of the character string; and
generating, for the character string, a token comprising the plurality of character values and the string length of the character string;
generating, by the at least one processor, a document signature for the digital document by:
arranging the plurality of tokens in an order according to frequency of occurrence in the digital document; and
combining one or more subsets of tokens of the plurality of tokens into a token sequence based on the order of the plurality of tokens; and
determining, by the at least one processor accessing the plurality of digital documents stored at the digital content database, a cluster of similar digital documents comprising the digital document by comparing the document signature to a plurality of document signatures corresponding to the plurality of digital documents.
19. The non-transitory computer readable storage medium of claim 18 , wherein generating the plurality of tokens comprises:
determining the plurality of character values of the character string by determining a plurality of sets of character values of a plurality of portions of the character string; and
concatenating the plurality of sets of character values and the string length within a token for the character string.
20. The non-transitory computer readable storage medium of claim 18 , wherein determining the cluster of similar digital documents comprises:
generating, by comparing document signatures of the plurality of document signatures, a similarity index comprising similarity scores for pairs of digital documents of the plurality of digital documents; and
determining the cluster of similar digital documents comprising the digital document in response to one or more similarity scores for pairs of digital documents comprising the digital document meeting a similarity threshold.