Granted Patent B2

US 8,914,331 · App. 14/148,713 · Granted Dec 16, 2014

Abstract

Claims (56)

1. A computer-implemented system for identifying duplicate and near duplicate messages, comprising:

a set of messages;

an extraction module to extract metadata from each of the messages;

a compiler to compile the messages into a master array based on the extracted metadata;

a topic module to determine topics of the messages in the master array;

a topic sorting module to sort the messages in the master array based on the topics;

a message selection module to select one of the messages and to compare the selected message with a next message;

a conversation module to determine that the selected and next messages are from a common conversation thread when the topics match;

a comparison module to compare a body of one such message with the body of each other message;

a duplicate identification module to identify those messages having matching bodies as exact duplicates and to remove the exact duplicate messages from the set;

a sort module to sort the remaining messages in order of message length;

a recursive determination module to compare a shorter message with a longer message and to determine that the body of the shorter message is included in the body of the longer message;

a near-duplicate module to mark the shorter message as a near-duplicate of the longer message; and

a processor to execute the modules.

2. A system according to claim 1 , further comprising:

a thread identifier to identify thread markers within one or more of the messages, wherein the thread markers comprise at least one of separators and subject line indicators.

3. A system according to claim 2 , further comprising:

an organization module to organize the remaining messages by the message length based on a number of thread markers.

4. A system according to claim 1 , further comprising:

a message marking module to determine that the selected message is the first message for the topic and to mark the first message as a beginning of a range for the matching topic.

5. A system according to claim 1 , further comprising:

a message marking module to select one of the messages, to compare the selected message with a next message, to determine that the topics of the selected message and the next message do not match, to determine that the selected message is a first message of the topic, and to mark the selected message as a unique message.

6. A system according to claim 5 , further comprising:

a log module to enter the unique message in a log and to cross-reference the unique message with any duplicate and near-duplicate messages of the unique message.

7. A system according to claim 5 , further comprising:

a forwarding module to forward the unique message to a further phase of a document review process.

8. A computer-implemented method for identifying duplicate and near duplicate messages, comprising:

obtaining a set of messages;

extracting metadata from each of the messages;

compiling the messages into a master array based on the extracted metadata;

determining topics of the messages in the master array;

sorting the messages in the master array based on the topics;

selecting one of the messages and comparing the selected message with a next message;

determining that the selected and next messages are from a common conversation thread when the topics match;

comparing a body of one such message with the body of each other message;

identifying those messages having matching bodies as exact duplicates and removing the exact duplicate messages from the set;

sorting the remaining messages in order of message length;

comparing a shorter message with a longer message and determining that the body of the shorter message is included in the body of the longer message; and

marking the shorter message as a near duplicate of the longer message.

9. A method according to claim 8 , further comprising:

identifying thread markers within one or more of the messages, wherein the thread markers comprise at least one of separators and subject line indicators.

10. A method according to claim 9 , further comprising:

organizing the remaining messages by the message length based on a number of thread markers.

11. A method according to claim 8 , further comprising:

determining that the selected message is the first message for the topic; and

marking the first message as a beginning of a range for the matching topic.

12. A method according to claim 8 , further comprising:

selecting one of the messages and comparing the selected message with a next message;

determining that the topics of the selected message and the next message do not match;

determining that the selected message is a first message of the topic; and

marking the selected message as a unique message.

13. A method according to claim 12 , further comprising:

entering the unique message in a log; and

cross-referencing the unique message with any duplicate and near-duplicate messages of the unique message.

14. A method according to claim 12 , further comprising:

forwarding the unique message to a further phase of a document review process.

Assignments (6)