Apparatus and method of implementing enhanced batch-mode active learning for technology-assisted review of documents

1. A method to implement a Cohen's kappa process to determine stopping criteria, comprising:

by one or more computing devices:

performing an iteration of active learning using a classification model M c ;

constructing an updated classification model M c (x) using extended training data documents D c ;

obtaining a current version of the updated classification model M c (x), an unlabeled set of available documents D and extended training data documents D c collectively referred to as a total set of documents D T , and a stopping threshold t stop ;

constructing a score vector S c using the current classification model M c (x) and the total set of documents D T ;

retrieving a previous score vector S c−1 ;

obtaining a set of documents D +c from the documents D T that have a positive score in S c ;

obtaining a set of documents D −c , from the documents D T that have a negative score in S c ;

obtaining a set of documents D +(c−1) from the documents D T that have a positive score in S c−1 ;

obtaining a set of documents D −(c−1) from the documents D T that have a negative score in S c−1 ;

obtaining a set of documents in common, D + , between D +c and D +(c−1) ;

obtaining a set of documents in common, D − , between D −c and D −(c−1) ;

obtaining a probability, P + , of a document having a positive score in both score vectors S c and S c−1 by counting a number of documents, N + , in D + divided by the total number of documents, N, in D;

obtaining a probability, P − , of a document having negative score in both score vectors S c and S c−1 , by counting a number of documents, N − , in D − divided by the total number of documents, N, in D;

obtaining a value A o as P + +P − ;

obtaining a probability, P +c , by counting a number of documents, N +c , in D +c divided by the number of documents, N, in D;

obtaining a probability, P −c , by counting a number of documents, N −c , in D −c divided by the number of documents, N, in D;

obtaining a probability, P +(c−1) , by counting a number of documents, N +(c−1) , in D +(c−1) divided by the number of documents, N, in D;

obtaining a probability, P −(c−1) , by counting a number of documents, N−( c−1) , in D −(c−1) divided by the number of documents, N, in D;

obtaining a value A e as a probability of obtaining a positive document, P +c *P +(c−1) , plus a probability of obtaining a negative document, P −c *P −(c−1) ;

obtaining a Kappa value L as A o A e divided by (1−A o ); and

comparing the Kappa value L to the stopping threshold t stop to determine whether t stop ≤A o −A e /1−A e which indicates that a stopping criteria has been met.

2. The method of claim 1 , further comprising:

based on the determination that the stopping criteria has not been met, storing the score vector S c to memory as a previous score vector S c−1 ; and

based on the determination that the stopping criteria has not been met, repeating each of the steps.

3. The method of claim 2 , further comprising, based on the determination that the stopping criteria has been met, returning the updated classification model M c (x).

4. The method of claim 1 , wherein the classification model M c (x) is a hyperplane.

5. The method of claim 1 , wherein the iteration of active learning is implemented using a support vector machine (SVM).

6. The method of claim 1 , wherein performing an iteration of active learning comprises:

(i) selecting a new batch of unlabeled instances B c using the current classification model M c (x), the unlabeled set of available documents D, and a batch size k;

(ii) obtaining labels for the new batch of unlabeled instances B c ; and

(iii) adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .

7. The method of claim 6 , wherein the selecting of the new batch of unlabeled instances B c , utilizes a biased probabilistic sampler process.

8. The method of claim 6 , wherein the selecting of the new batch of unlabeled instances B c , utilizes a diversity sampler process.

9. The method of claim 1 , wherein the method is implemented in a technology-assisted document review.

10. An apparatus configured to implement a Cohen's kappa process to determine stopping criteria, the apparatus comprising:

a processor; and,

a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions, whereby the processor is operable to:

perform an iteration of active learning using a classification model M c ;

construct an updated classification model M c (x) using extended training data documents D c ;

obtain a current version of the updated classification model M c (x), an unlabeled set of available documents D and extended training data documents D c collectively referred to as a total set of documents D T , and a stopping threshold t stop ;

construct a score vector S c using the current classification model M c (x) and the total set of documents D T ;

retrieve a previous score vector S c−1 ;

obtain a set of documents D +c from the documents D T that have a positive score in S c ;

obtain a set of documents D −c from the documents D T that have a negative score in S c ;

obtain a set of documents D +(c−1) from the documents D T that have a positive score in S c−1 ;

obtain a set of documents D −(c−1) from the documents D T that have a negative score in S c−1 ;

obtain a set of documents in common, D + , between D +c and D +(c−1) ;

obtain a set of documents in common, D − , between D −c and D −(c−1) ;

obtain a probability, P + , of a document having a positive score in both score vectors S c and S c−1 by counting a number of documents, N + , in D + divided by the total number of documents, N, in D;

obtain a probability, P − , of a document having negative score in both score vectors S c and S c−1 , by counting a number of documents, N − , in D − divided by the total number of documents, N, in D;

obtain a value A o as P + +P − ;

obtain a probability, P +c , by counting a number of documents, N +c , in D +c divided by the number of documents, N, in D;

obtain a probability, P −c , by counting a number of documents, N −c , in D −c divided by the number of documents, N, in D;

obtain a probability, P +(c−1) , by counting a number of documents, N +(c−1) , in D +(c−1) divided by the number of documents, N, in D;

obtain a probability, P −(c−1) , by counting a number of documents, N−( c−1) , in D −(c−1) divided by the number of documents, N, in D;

obtain a value A e as a probability of obtaining a positive document, P +c *P +(c−1) , plus a probability of obtaining a negative document, P −c *P −(c−1) ;

obtain a Kappa value L as A o −A e divided by (1−A o ); and

compare the Kappa value L to the stopping threshold t stop to determine whether t stop ≤A o −A e /1−A e which indicates that a stopping criteria has been met.

11. The apparatus of claim 10 , wherein the processor is further operable to:

based on the determination that the stopping criteria has not been met, store the score vector S c to memory as a previous score vector S c−1 ; and

based on the determination that the stopping criteria has not been met, repeat each of the steps.

12. The apparatus of claim 11 , further comprising, based on the determination that the stopping criteria has been met, returning the updated classification model M c (x).

13. The apparatus of claim 10 , wherein the classification model M c (x) is a hyperplane.

14. The apparatus of claim 10 , wherein the iteration of active learning is implemented using a support vector machine (SVM).

15. The apparatus of claim 10 , wherein performing an iteration of active learning comprises:

(i) selecting a new batch of unlabeled instances B c using the current classification model M c (x), the unlabeled set of available documents D, and a batch size k;

(ii) obtaining labels for the new batch of unlabeled instances B c ; and

(iii) adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .

16. The apparatus of claim 15 , wherein the selecting of the new batch of unlabeled instances B c , utilizes a biased probabilistic sampler process.

17. The apparatus of claim 15 , wherein the selecting of the new batch of unlabeled instances B c , utilizes a diversity sampler process.

18. The apparatus of claim 10 , wherein the apparatus is implemented in a technology-assisted document review process.

19. A computer program product, comprising:

a non-transitory computer-readable storage device having computer-executable program instructions embodied thereon that when executed by a computer cause the computer to implement a Cohen's kappa process to determine stopping criteria, the computer-executable program instructions comprising computer-executable program instructions to:

perform an iteration of active learning using a classification model M c ;

construct an updated classification model M c (x) using extended training data documents D c ;

obtain a current version of the updated classification model M c (x), an unlabeled set of available documents D and extended training data documents D c collectively referred to as a total set of documents D T , and a stopping threshold t stop ;

construct a score vector S c using the current classification model M c (x) and the total set of documents D T ;

retrieve a previous score vector S c−1 ;

obtain a set of documents D +c from the documents D T that have a positive score in S c ;

obtain a set of documents D −c from the documents D T that have a negative score in S c ;

obtain a set of documents D +(c−1) from the documents D T that have a positive score in S c−1 ;

obtain a set of documents D −(c−1) from the documents D T that have a negative score in S c−1 ;

obtain a set of documents in common, D + , between D +c and D +(c−1) ;

obtain a set of documents in common, D − , between D −c and D −(c−1) ;

obtain a probability, P + , of a document having a positive score in both score vectors S c and S c−1 by counting a number of documents, N + , in D + divided by the total number of documents, N, in D;

obtain a probability, P − , of a document having negative score in both score vectors S c and S c−1 , by counting a number of documents, N − , in D − divided by the total number of documents, N, in D;

obtain a value A o as P + +P − ;

obtain a probability, P +c , by counting a number of documents, N +c , in D +c divided by the number of documents, N, in D;

obtain a probability, P −c , by counting a number of documents, N −c , in D −c divided by the number of documents, N, in D;

obtain a probability, P +(c−1) , by counting a number of documents, N +c−1 , in D +(c−1) divided by the number of documents, N, in D;

obtain a probability, P −(c−1) , by counting a number of documents, N−( c−1) , in D −(c−1) divided by the number of documents, N, in D;

obtain a value A e as a probability of obtaining a positive document, P +c *P +(c−1) , plus a probability of obtaining a negative document, P −c *P −(c−1) ;

obtain a Kappa value L as A o −A e divided by (1−A o ); and

compare the Kappa value L to the stopping threshold t stop to determine whether t stop ≤A o −A e /1−A e which indicates that a stopping criteria has been met.

20. The computer program product of claim 19 ; further comprising:

based on the determination that the stopping criteria has not been met, store the score vector S c to memory as a previous score vector S c−1 ; and

based on the determination that the stopping criteria has not been met, repeat each of the steps.

21. The computer program product of claim 20 , further comprising, based on the determination that the stopping criteria has been met, returning the updated classification model M c (x).

22. The computer program product of claim 19 , wherein the classification model M c (x) is a hyperplane.

23. The computer program product of claim 19 , wherein the iteration of active learning is implemented using a support vector machine (SVM).

24. The computer program product of claim 19 , wherein performing an iteration of active learning comprises:

(i) selecting a new batch of unlabeled instances B c using the current classification model M c (x), the unlabeled set of available documents D, and a batch size k;

(ii) obtaining labels for the new batch of unlabeled instances B c ; and

(iii) adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .

25. The computer program product of claim 24 , wherein the selecting of the new batch of unlabeled instances B c , utilizes a biased probabilistic sampler process.

26. The computer program product of claim 24 , wherein the selecting of the new batch of unlabeled instances B c , utilizes a diversity sampler process.

27. The computer program product of claim 19 , wherein the computer program product is implemented in a technology-assisted document review process.