Methods and systems to efficiently find similar and near-duplicate emails and files
View Patent ↗A set of trigrams can be generated for each document in a plurality of documents processed by an e-discovery system. Each trigram in the set of trigrams for a given document is a sequence of three terms in the given document. A set of trigrams for each similar document is then determined based on the set of trigrams for the original document. To facilitate identification of the similar documents, a full text index is then generated for the plurality of documents and the set of trigrams for each document are indexed into the full text index, as individual terms. Queries can be generated into the full text index based on trigrams of a document to determine other similar or near-duplicate documents. After a set of potentially similar documents are identified, a separate distance criteria can be applied to evaluate the level of similarity between the two documents in an efficient way.
1. A method comprising:
receiving a plurality of documents, each document in the plurality of documents having a plurality of terms;
for each document in the plurality of documents:
generating, by a processor, a first set of trigrams, each trigram in the first set of trigrams being a sequence of terms, and
determining a second set of trigrams based on the first set of trigrams for the document and first filter criteria to filter the first set of trigrams for the document for a predetermined number of most frequently occurring trigrams, the second set of trigrams being a smaller subset of the first set of trigrams for the document, the second set of trigrams comprising a trigram that satisfies the first filter criteria; and
generating a full text index indexing the second set of trigrams for the plurality of documents;
generating a query into the full text index for the plurality of documents indexing the second set of trigrams for each document in the plurality of documents based on a third set of trigrams associated with a new document; and
identifying, in response to the query into the full text index, a set of documents in the plurality of documents that are substantially similar to the new document by identifying matching trigrams between the third set of trigrams and the second set of trigrams indexed in the full text index.
2. The method of claim 1 , further comprising:
for each document in the plurality of documents, generating an artificial distance between a trigram in the second set of trigrams prior to indexing into the full text index.
3. The method of claim 1 , further comprising:
receiving the new document;
determining the third set of trigrams associated with the new document.
4. The method of claim 1 wherein identifying the set of documents in the plurality of documents that are substantially similar to the new document comprises identifying the set of documents based on second filter criteria.
5. The method of claim 1 wherein identifying the set of documents in the plurality of documents that are substantially similar to the new document comprises identifying the set of documents as a subset of another set of documents returned in response to executing the query based on a cosine similarity between the new document and each document in the other set of documents.
6. The method of claim 1 , further comprising:
generating a user interface to display information identifying the set of documents in the plurality of documents as substantially similar to the new document in response to the query into the full text index for the plurality of documents indexing the second set of trigrams based on the third set of trigrams associated with the new document.
7. The method of claim 1 , further comprising:
generating a user interface to display information identifying a first document of the plurality of documents as substantially similar to the new document and to display a part of the first document of the plurality of documents corresponding to a match with the new document.
8. A non-transitory computer-readable medium storing instructions, which when executed, cause a processor to perform operations comprising:
receiving a plurality of documents, each document in the plurality of documents having a plurality of terms;
for each document in the plurality of documents:
generating, by the processor, a first set of trigrams, each trigram in the first set of trigrams being a sequence of terms, and
determining a second set of trigrams based on the first set of trigrams for the document and first filter criteria to filter the first set of trigrams for the document for a predetermined number of most frequently occurring trigrams, the second set of trigrams being a smaller subset of the first set of trigrams for the document, the second set of trigrams comprising a trigram that satisfies the first filter criteria; and
generating a full text index indexing the second set of trigrams for the plurality of documents;
generating a query into the full text index for the plurality of documents indexing the second set of trigrams for each document in the plurality of documents based on a third set of trigrams associated with a new document; and
identifying, in response to the query, a set of documents in the plurality of documents that are substantially similar to the new document by identifying matching trigrams between the third set of trigrams and the second set of trigrams indexed in the full text index.
9. The computer-readable medium of claim 8 , further comprising:
generating an artificial distance between a trigram in the second set of trigrams prior to indexing into the full text index.
10. The computer-readable medium of claim 8 , further comprising:
receiving the new document;
determining the third set of trigrams associated with the new document.
11. The computer-readable medium of claim 8 wherein identifying the set of documents in the plurality of documents that are substantially similar to the new document comprises identifying the set of documents based on second filter criteria.
12. The computer-readable medium of claim 8 wherein identifying the set of documents in the plurality of documents that are substantially similar to the new document comprises identifying the set of documents as a subset of another set of documents returned in response to executing the query based on a cosine similarity between the new document and each document in the other set of documents.
13. The computer-readable medium of claim 8 , further comprising:
generating a user interface to display information identifying a first document of the plurality of documents as substantially similar to the new document selected in response to the query into the full text index for the plurality of documents indexing the second set of trigrams for each document in the plurality of documents based on the third set of trigrams associated with the new document.
14. The computer-readable medium of claim 8 , further comprising:
generating a user interface to display information identifying a first document of the plurality of documents as substantially similar to the new document and to display a part of the first document of the plurality of documents corresponding to a match with the new document.
15. A system comprising:
a memory; and
a processor coupled to the memory, wherein the processor is to:
receive a plurality of documents, each document in the plurality of documents having a plurality of terms;
for each document in the plurality of documents:
generate a first set of trigrams, each trigram in the first set of trigrams being a sequence of terms, and
determine a second set of trigrams based on the first set of trigrams for the document and first filter criteria to filter the first set of trigrams for the document for a predetermined number of most frequently occurring trigrams, the second set of trigrams being a smaller subset of the first set of trigrams for the document, the second set of trigrams comprising a trigram that satisfies the first filter criteria; and
generate a full text index indexing the second set of trigrams for the plurality of documents;
generate a query into the full text index for the plurality of documents indexing the second set of trigrams for each document in the plurality of documents based on a third set of trigrams associated with a new document; and
identify, in response to the query, a set of documents in the plurality of documents that are substantially similar to the new document by identifying matching trigrams between the third set of trigrams and the second set of trigrams indexed in the full text index.
16. The system of claim 15 wherein the processor is further to:
receive the new document;
determine the third set of trigrams associated with the new document;
generate a user interface to display information identifying selected ones of the set of documents as substantially similar to the new document;
generate an artificial distance between a trigram in the second set of trigrams prior to indexing into the full text index; and
generate a user interface to display a part of the new document that corresponds to a match with the new document.