System for enhancing expert-based computerized analysis of a set of digital documents and methods useful in conjunction therewith

1. An electronic document analysis method receiving N electronic documents pertaining to a case encompassing a set of issues including at least one issue and establishing relevance of at least the N electronic documents to at least one individual issue in the set of issues, the method performed with a processor, the method comprising, for at least one individual issue from among said set of issues:

i. receiving an output of a categorization process applied to documents in at least control subsets of said at least N electronic documents, said output including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a nonrelevant-to-said-individual issue indication;

ii. seeking an input as to whether or not to initiate a new iteration I; if a new iteration is initiated, perform steps iv-x;

iii. if a new iteration is not initiated, go to step xi;

iv. receiving an output of a categorization process applied to the m electronic documents;

v. adding m electronic documents to an existing training subset and building a text classifier simulating said categorization process using said output for all documents in said training subset of documents;

vi. evaluating said text classifier's quality using said output for documents in said control subset;

vii. selecting a cut-off point for binarizing said rankings of said documents in said control subset;

viii. using said cut-off point, computing and storing at least one quality criterion characterizing said binarizing of said rankings of said documents in said control subset, thereby to define a quality of performance indication of a current iteration I;

ix. displaying a comparison of the quality of performance indication of the current iteration I to quality of performance indications of previous iterations;

x. returning to step ii; and

xi. generating a computer display of said output of said categorization process received in step i as most recently performed, including, for each document in said subsets, one of a relevant-to-said-individual issue indication and a non-relevant-to-said-individual issue indication.

2. The method according to claim 1 wherein said receiving comprises receiving an output of a categorization process performed by a human operator.

3. The method according to claim 1 wherein said cut-off point is selected from all ranks of documents in said control subset so as to maximize a quality criterion.

4. The method according to claim 1 wherein said displaying a comparison comprises generating at least one graph of at least one quality criterion vs. iteration serial number.

5. The method according to claim 1 wherein said input comprises a user input received from a human user.

6. The method according to claim 4 wherein said input comprises a computerized input including a computerized indication of flatness of said graph of at least one quality criterion vs. iteration serial number.

7. The method of claim 1 , the method also comprising:

performing a plurality of machine learning iterations on said N electronic documents wherein said iterations determine relevance of documents to at least one issue in said set of issues;

determining at least one relevance determination quality criterion characterizing current relevance determination performance; and

estimating a cost effectiveness of continued iterations on said at least N electronic documents vs. termination thereof.

8. The method according to claim 7 wherein said estimating includes estimating at least one relevance determination quality criterion of future relevance determination performance assuming continued iterations.

9. The method according to claim 7 wherein said estimating includes computing a budget required to enable continued iterations.

10. The method according to claim 9 wherein said culling percentage is computed by:

generating a graph of a relevance determination quality criterion characterizing the iterative apparatus's performance during an iteration as a function of a serial number of said iteration; and

computing an integral thereof.

11. A method according to claim 1 , the method also comprising:

generating binary relevance data characterizing relevance of said documents to at least one issue in said set of issues, said binary relevance data being generated by applying a cut-off point to multi-value relevance data; and

estimating the relative cost effectiveness of a multiplicity of possible cut-off points, including:

generating a computer display of a continuum of possible cut-off points, each position along said continuum corresponding to a possible cut-off point, accepting a user's indication of positions along said continuum, and computing and displaying, for each user-indicated position, cost effectiveness information characterizing the cut-off point corresponding to said user-indicated position.

12. The method according to claim 1 wherein said computer display of said final output comprises a histogram of ranks for each issue.

13. The method according to claim 1 wherein said computer display of said final output comprises a function of an indication of a quality measure for each of a plurality of cut-off points.

14. The method according to claim 13 wherein said quality measure is selected from the following group: un-weighted F-measure; weighted F-measure; precision; recall; and accuracy.

15. The method according to claim 13 wherein said function of said indication of a quality measure for each of a plurality of cut-off points comprises a graph of said quality measure as a function of cut-off point.

16. The method according to claim 15 wherein said function of said indication comprises a culling percentage including an integral of said graph.

17. The method according to claim 1 wherein said set of issues comprises a plurality of issues and said computer display includes an indication of documents relevant to a logical combination of a plurality of issues.

18. The method according to claim 1 further comprising using said text classifier most recently built to generate, for at least one individual issue in said set of issues, a set of keywords differentiating documents relevant to said individual issue to documents irrelevant to said individual issue.

19. The method according to claim 7 wherein said set of issues comprises a plurality of issues and also comprising monitoring the system's analysis of relevance of said at least N electronic documents to each of the plurality of issues and prompting at least one of the user and the system if relevance has not been analyzed for some of said plurality of issues.

20. A method according to claim I comprising:

generating a computer display of at least one user-selected document within said set of documents, wherein at least some words in said user-selected document are differentially presented depending on their contribution to said classification of the document as relevant or irrelevant by said text classifier;

sequentially removing certain sets of words from each individual document in the set of documents and using the text classifier to classify said document's relevance assuming said words are removed, thereby to obtain a relevance output for each set of words; and

comparing said relevance output to an output obtained by using the text classifier to classify said individual document without removing any words, thereby to obtain an indication of the contribution of each set of words to the relevance of the document.