System and method for near and exact de-duplication of documents
A system, method and computer program product for identifying near and exact-duplicate documents in a document collection, including for each document in the collection, reading textual content from the document; filtering the textual content based on user settings; determining N most frequent words from the filtered textual content of the document; performing a quorum search of the N most frequent words in the document with a threshold M; and sorting results from the quorum search based on relevancy. Based on the values of N and M near and exact-duplicate documents are identified in the document collection.
1. A method for identifying near and exact-duplicate documents in a document collection, the method comprising:
for each document in the collection performing by a computer the steps of:
reading textual content from the document;
filtering the textual content based on user settings;
determining N most frequent words from the filtered textual content of the document to generate a first most frequent word, word- 1 , to an Nth most frequent word, word-N, sorted from highest to lowest frequency;
performing a quorum search using the N most frequent words from the filtered textual content in the document with a threshold M, wherein the threshold M is used to retrieve documents from the document collection having a number M of the N most frequent words;
sorting results from the quorum search based on relevancy,
whereby based on the values of N and M near and exact-duplicate documents are identified in the document collection; and
calculating M based on a precision value percentage user setting by multiplying the percentage value times the N value.
2. The method of claim 1 , further comprising associating a respective XML wrapper for each document in the collection, wherein the XML wrapper includes a unique document identification for the document, and unique document identifications for near and exact-duplicate documents of the document.
3. The method of claim 2 , wherein the reading of the textual content from the document includes reading the XML wrapper for the document.
4. The method of claim 1 , further comprising reading user preferences for the values of N and M, and including content filtering settings for the filtering of the textual content.
5. The method of claim 4 , wherein the content filtering settings include filtering of numbers, keyfields, noise words, and optical character recognition errors.
6. The method of claim 1 , wherein the reading of the textual content from the document includes reading user settings to determine a text to read from the document.
7. The method of claim 6 , wherein the user settings to determine the text to read from the document include settings for reading the entire text from the document, reading only a first number of Kb from the document, reading random sections of text from the document, and reading only a first number of Kb after a key phrase or key word from the document.
8. The method of claim 1 , further comprising calculating N based on a recall percentage value user setting by multiplying the percentage value times a number of total words in the filtered text.
9. The method of claim 1 , further comprising calculating N based on a recall percentage value user setting by multiplying the percentage value times a number of unique words in the filtered text.
10. The method of claim 1 , further comprising determining the relevancy by taking a number of hits for the quorum search in the document and dividing the number of hits by a size of the document in kilobytes of text in the document or a size in kilobytes for the entire document.
11. A computer program product for identifying near and exact-duplicate documents in a document collection and including one or more computer readable instructions embedded on a non-transitory computer readable medium and configured to cause one or more computer processors to perform the steps of:
for each document in the collection:
reading textual content from the document;
filtering the textual content based on user settings;
determining N most frequent words from the filtered textual content of the document to generate a first most frequent word, word- 1 , to an Nth most frequent word, word-N, sorted from highest to lowest frequency;
performing a quorum search using the N most frequent words from the filtered textual content in the document with a threshold M, wherein the threshold M is used to retrieve documents from the document collection having a number M of the N most frequent words;
sorting results from the quorum search based on relevancy,
whereby based on the values of N and M near and exact-duplicate documents are identified in the document collection; and
calculating M based on a precision value percentage user setting by multiplying the percentage value times the N value.
12. The computer program product of claim 11 , further comprising associating a respective XML wrapper for each document in the collection, wherein the XML wrapper includes a unique document identification for the document, and unique document identifications for near and exact-duplicate documents of the document.
13. The method of claim 12 , wherein the reading of the textual content from the document includes reading the XML wrapper for the document.
14. The computer program product of claim 11 , further comprising reading user preferences for the values of N and M, and including content filtering settings for the filtering of the textual content.
15. The method of claim 14 , wherein the content filtering settings include filtering of numbers, keyfields, noise words, and optical character recognition errors.
16. The computer program product of claim 11 , wherein the reading of the textual content from the document includes reading user settings to determine a text to read from the document.
17. The method of claim 16 , wherein the user settings to determine the text to read from the document include settings for reading the entire text from the document, reading only a first number of Kb from the document, reading random sections of text from the document, and reading only a first number of Kb after a key phrase or key word from the document.
18. The computer program product of claim 11 , further comprising calculating N based on a recall percentage value user setting by multiplying the percentage value times a number of total words in the filtered text.
19. The computer program product of claim 11 , further comprising calculating N based on a recall percentage value user setting by multiplying the percentage value times a number of unique words in the filtered text.
20. The computer program product of claim 11 , further comprising determining the relevancy by taking a number of hits for the quorum search in the document and dividing the number of hits by a size of the document in kilobytes of text in the document or a size in kilobytes for the entire document.
21. A system for identifying near and exact-duplicate documents in a document collection, the system comprising:
a computer configured to perform for each document in the collection the steps of:
reading textual content from the document;
filtering the textual content based on user settings;
determining N most frequent words from the filtered textual content of the document to generate a first most frequent word, word- 1 , to an Nth most frequent word, word-N, sorted from highest to lowest frequency;
performing a quorum search using the N most frequent words from the filtered textual content in the document with a threshold M, wherein the threshold M is used to retrieve documents from the document collection having a number M of the N most frequent words;
sorting results from the quorum search based on relevancy,
whereby based on the values of N and M near and exact-duplicate documents are identified in the document collection; and
calculating M based on a precision value percentage user setting by multiplying the percentage value times the N value.
22. The system of claim 21 , wherein the computer is further configured for associating a respective XML wrapper for each document in the collection, wherein the XML wrapper includes a unique document identification for the document, and unique document identifications for near and exact-duplicate documents of the document.
23. The method of claim 22 , wherein the reading of the textual content from the document includes reading the XML wrapper for the document.
24. The system of claim 21 , wherein the computer is further configured for reading user preferences for the values of N and M, and including content filtering settings for the filtering of the textual content.
25. The method of claim 24 , wherein the content filtering settings include filtering of numbers, keyfields, noise words, and optical character recognition errors.
26. The system of claim 21 , wherein the reading of the textual content from the document includes reading user settings to determine a text to read from the document.
27. The method of claim 26 , wherein the user settings to determine the text to read from the document include settings for reading the entire text from the document, reading only a first number of Kb from the document, reading random sections of text from the document, and reading only a first number of Kb after a key phrase or key word from the document.
28. The system of claim 21 , wherein the computer is further configured for calculating N based on a recall percentage value user setting by multiplying the percentage value times a number of total words in the filtered text.
29. The system of claim 21 , wherein the computer is further configured for calculating N based on a recall percentage value user setting by multiplying the percentage value times a number of unique words in the filtered text.
30. The system of claim 21 , wherein the computer is further configured for determining the relevancy by taking a number of hits for the quorum search in the document and dividing the number of hits by a size of the document in kilobytes of text in the document or a size in kilobytes for the entire document.