Method and system for information retrieval effectiveness estimation in e-discovery
A server computing system determines a plurality of statistics for a plurality of test documents, determines a number of false negatives for a corpus of documents based on one or more of the plurality of statistics for the plurality of test documents. The classification of a document of the corpus of documents is a false negative if classification of the document by a classification model is negative and classification of the document by a user is positive. The server computing system calculates an effectiveness of an information retrieval system on a corpus of documents based on the number of false negatives for the corpus of documents.
1. A method for estimating the effectiveness of information retrieval for electronic discovery comprising:
calculating, by at least one computer processor configured to operate in an information retrieval system, a plurality of statistics for a plurality of test documents, wherein the plurality of statistics for the plurality of test documents comprises a number of documents that are false negatives in the plurality of test documents;
calculating, by the at least one computer processor, the number of false negatives for a corpus of documents based on one or more of a number of test documents in the plurality of test documents, a size of the corpus of documents, a predetermined confidence level, and the number of false negatives in the plurality of test documents, wherein classification of a document of the corpus of documents is a false negative if classification of the document by a classification model is negative and classification of the document by a user is positive; and
calculating, by the at least one computer processor, an effectiveness of the information retrieval system on the corpus of documents based on the number of false negatives for the corpus of documents.
2. The method of claim 1 , wherein the plurality of statistics for the plurality of test documents further comprises at least one of a number of documents that are true positives in the plurality of test documents or a number of documents that are false positives in the plurality of test documents.
3. The method of claim 1 , wherein the plurality of statistics for the plurality of test documents further comprises a number of documents that are true positives in the plurality of test documents and a number of documents that are false positives in the plurality of test documents, and calculating the effectiveness of the information retrieval system further comprises:
calculating a number of true positives in the corpus of documents based on the number of test documents in the plurality of test documents, the size of the corpus of documents, the predetermined confidence level, and the number of documents that are true positives in the plurality of test documents;
calculating a number of false positives in the corpus of documents based on the number of test documents in the plurality of test documents, the size of the corpus of documents, the predetermined confidence level, and the number of false positives in the plurality of test documents; and
calculating one or more effectiveness measures for the corpus of documents based on the number of false negatives in the corpus of documents and at least one of the number of true positives in the corpus of documents or the number of false positives in the corpus of documents.
4. The method of claim 3 , wherein the effectiveness measures comprise recall and F-measure.
5. The method of claim 1 , wherein determining the plurality of statistics for the plurality of test documents comprises:
obtaining a user classification for each of the plurality of test documents from a user;
obtaining a system classification for each of the plurality of test documents from an information retrieval system; and
calculating the plurality of statistics for the plurality of test documents based on the user classification for each of the plurality of test documents and the system classification for each of the plurality of test documents.
6. The method of claim 1 , further comprising:
determining the number of test documents to be included in the plurality of test documents based on a plurality of statistics for a plurality of validation documents and an effectiveness measure of interest.
7. The method of claim 6 , wherein determining the plurality of statistics for the plurality of validation documents comprises:
obtaining a user classification for each of the plurality of validation documents from a user;
obtaining a system classification for each of the plurality of validation documents from the information retrieval system; and
calculating the plurality of statistics for the plurality of validation documents based on the user classification for each of the plurality of validation documents and the system classification for each of the plurality of validation documents.
8. A non-transitory computer-readable storage medium having instructions that, when executed by at least one computer processor configured to operate in an information retrieval system, cause the at least one computer processor to perform operations to estimate the effectiveness of information retrieval for electronic discovery comprising:
calculating, by the at least one computer processor, a plurality of statistics for a plurality of test documents, wherein the plurality of statistics for the plurality of test documents comprises a number of documents that are false negatives in the plurality of test documents;
calculating, by the at least one computer processor, the number of false negatives for a corpus of documents based on one or more of a number of test documents in the plurality of test documents, a size of the corpus of documents, a predetermined confidence level, and the number of false negatives in the plurality of test documents, wherein classification of a document of the corpus of documents is a false negative if classification of the document by a classification model is negative and classification of the document by a user is positive; and
calculating, by the at least one computer processor, an effectiveness of the information retrieval system on the corpus of documents based on the number of false negatives for the corpus of documents.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the plurality of statistics for the plurality of test documents further comprises at least one of a number of documents that are true positives in the plurality of test documents or a number of documents that are false positives in the plurality of test documents.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the plurality of statistics for the plurality of test documents further comprises a number of documents that are true positives in the plurality of test documents and a number of documents that are false positives in the plurality of test documents, and calculating the effectiveness of the information retrieval system further comprises:
calculating a number of true positives in the corpus of documents based on the number of test documents in the plurality of test documents, the size of the corpus of documents, the predetermined confidence level, and the number of documents that are true positives in the plurality of test documents;
calculating a number of false positives in the corpus of documents based on the number of test documents in the plurality of test documents, the size of the corpus of documents, the predetermined confidence level, and the number of false positives in the plurality of test documents; and
calculating one or more effectiveness measures for the corpus of documents based on the number of false negatives in the corpus of documents and at least one of the number of true positives in the corpus of documents or the number of false positives in the corpus of documents.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the effectiveness measures comprise recall and F-measure.
12. The non-transitory computer-readable storage medium of claim 8 , wherein determining the plurality of statistics for the plurality of test documents comprises:
obtaining a user classification for each of the plurality of test documents from a user;
obtaining a system classification for each of the plurality of test documents from an information retrieval system; and
calculating the plurality of statistics for the plurality of test documents based on the user classification for each of the plurality of test documents and the system classification for each of the plurality of test documents.
13. The non-transitory computer-readable storage medium of claim 8 , the operations further comprising:
determining the number of test documents to be included in the plurality of test documents based on a plurality of statistics for a plurality of validation documents and an effectiveness measure of interest.
14. The non-transitory computer-readable storage medium of claim 13 , wherein determining the plurality of statistics for the plurality of validation documents comprises:
obtaining a user classification for each of the plurality of validation documents from a user;
obtaining a system classification for each of the plurality of validation documents from the information retrieval system; and
calculating the plurality of statistics for the plurality of validation documents based on the user classification for each of the plurality of validation documents and the system classification for each of the plurality of validation documents.
15. A system for estimating the effectiveness of information retrieval for electronic discovery comprising:
a memory; and
at least one computer processor coupled to the memory, wherein the at least one computer processor is configured to operate in an information retrieval system to:
calculate a plurality of statistics for a plurality of test documents wherein the plurality of statistics for the plurality of test documents comprises a number of documents that are false negatives in the plurality of test documents;
calculate the number of false negatives for a corpus of documents based on one or more of the a number of test documents in the plurality of test documents, a size of the corpus of documents, a predetermined confidence level, and the number of false negatives in the plurality of test documents, wherein classification of a document of the corpus of documents is a false negative if classification of the document by a classification model is negative and classification of the document by a user is positive; and
calculate an effectiveness of the information retrieval system on the corpus of documents based on the number of false negatives for the corpus of documents.
16. The system of claim 15 , wherein the plurality of statistics for the plurality of test documents further comprises at least one of a number of documents that are true positives in the plurality of test documents or a number of documents that are false positives in the plurality of test documents.
17. The system of claim 15 , wherein the plurality of statistics for the plurality of test documents further comprises a number of documents that are true positives in the plurality of test documents and a number of documents that are false positives in the plurality of test documents, and to calculate the effectiveness of the information retrieval system the at least one computer processor is further configured to:
calculate a number of true positives in the corpus of documents based on the number of test documents in the plurality of test documents, the size of the corpus of documents, the predetermined confidence level, and the number of documents that are true positives in the plurality of test documents;
calculate a number of false positives in the corpus of documents based on the number of test documents in the plurality of test documents, the size of the corpus of documents, the predetermined confidence level, and the number of false positives in the plurality of test documents; and
calculate one or more effectiveness measures for the corpus of documents based on the number of false negatives in the corpus of documents and at least one of the number of true positives in the corpus of documents or the number of false positives in the corpus of documents.
18. The system of claim 17 , wherein the effectiveness measures comprise recall and F-measure.
19. The system of claim 15 , wherein to determine the plurality of statistics for the plurality of test documents, the processing device is to:
obtain a user classification for each of the plurality of test documents from a user; obtain a system classification for each of the plurality of test documents from an information retrieval system; and
calculate the plurality of statistics for the plurality of test documents based on the user classification for each of the plurality of test documents and the system classification for each of the plurality of test documents.
20. The system of claim 15 , wherein the processing device is further to:
determine the number of test documents to be included in the plurality of test documents based on a plurality of statistics for a plurality of validation documents and an effectiveness measure of interest.