Apparatus and method of implementing batch-mode active learning for technology-assisted review of documents

1. A method to implement a diversity sampler process to select new batches of unlabeled instances, comprising:

by one or more computing devices:

obtaining a current version of a classification model M, an unlabeled set of available documents D, and a cosine similarity threshold t;

sorting the unlabeled set of available documents D based on each of the documents absolute distance from the current version of the classification model M to obtain sorted indices I for each document of the unlabeled set of available documents D;

inserting the sorted document, of the sorted unlabeled set of available documents D, having a nearest sorted index I[1] from the current version of the classification model M into a new batch of unlabeled instances B c ;

obtaining sorted indices I of the sorted unlabeled set of available documents D that have a cosine angle ≥t with respect to the inserted document I[1];

removing the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D; and

repeating the insert operation, the second obtain operation, and the remove operation until the new batch of unlabeled instances B c are selected.

2. The method of claim 1 , wherein the classification model M is a hyperplane.

3. The method of claim 1 , wherein the unlabeled set of available documents D are sorted in increasing order.

4. The method of claim 1 , wherein, in the first obtaining step, the one or more computing devises further obtains a batch size k.

5. The method of claim 4 , wherein the one or more computing devises performs the steps of repeating the insert operation, the second obtain operation, and the remove operation until k documents are inserted into the new batch of unlabeled instances B c .

6. The method of claim 1 , wherein the Diversity Sampler process is implemented using a support vector machine (SVM).

7. The method of claim 1 , wherein the Diversity Sampler process is implemented in a technology-assisted document review.

8. The method of claim 1 , wherein the current version of the classification model M is created by:

obtaining an unlabeled set of documents D;

obtaining a batch size k;

constructing a first batch of k documents D;

obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and

constructing the current version of the classification model M using the training documents.

9. The method of claim 8 , further comprising:

obtaining labels for the new batch of unlabeled instances B c ; and

adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .

10. The method of claim 9 , further comprising constructing an updated classification model M using the extended training data documents D c .

11. A system configured to implement a diversity sampler process to select new batches of unlabeled instances, the apparatus comprising:

a processor; and

a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions, whereby the system is operable to:

obtain a current version of a classification model M, an unlabeled set of available documents D, and a cosine similarity threshold t;

sort the unlabeled set of available documents D based on each of the documents absolute distance from the current version of the classification model M to obtain sorted indices I for each document of the unlabeled set of available documents D;

insert the sorted document, of the sorted unlabeled set of available documents D, having a nearest sorted index I[1] from the current version of the classification model M into a new batch of unlabeled instances B c ;

obtain sorted indices I of the sorted unlabeled set of available documents D that have a cosine angle ≥t with respect to the inserted document I[1];

remove the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D; and

repeat the insert operation, the second obtain operation, and the remove operation until the new batch of unlabeled instances B c are selected.

12. The system of claim 11 , wherein the diversity sampler process is implemented using a support vector machine (SVM).

13. The system of claim 11 , wherein the current version of the classification model M is created by:

obtaining an unlabeled set of documents D;

obtaining a batch size k;

constructing a first batch of k documents D;

obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and

constructing the current version of the classification model M using the training documents.

14. The system of claim 13 , wherein the system is further operable to:

obtain labels for the new batch of unlabeled instances B c ; and

add the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .

15. The system of claim 14 , wherein the system is further operable to construct an updated classification model M using the extended training data documents D c .

16. A method to implement a biased probabilistic sampler process to select new batches of unlabeled instances, comprising:

by one or more computing devices:

obtaining a current version of a classification model M, an unlabeled set of available documents D, and a cosine similarity threshold t;

constructing a weight vector w based on an inverse operation of a distance from the current version of the classification model M for each document from the unlabeled set of available documents D;

normalizing the weight vector w for each document to convert the weight vector w into a probability vector;

choosing a document I[1] from the unlabeled set of available documents D using the corresponding probability vector;

inserting the chosen document I[1] into a new batch of unlabeled instances B c ;

identifying documents of the remaining unlabeled set of available documents D that have a cosine similarity angle ≥t with respect to the chosen document I[1];

removing the identified documents from the unlabeled set of available documents D;

re-normalizing the weight vector w for each remaining document in the set of documents D; and

repeating the choose operation, the insert operation, the identify operation, the remove operation, and the re-normalize operation until the new batch of unlabeled instances B c are selected.

17. The method of claim 16 , wherein, in the obtaining step, the one or more computing devices further obtains a batch size k.

18. The method of claim 17 , wherein the one or more computing devises performs the steps of repeating the choose operation, the insert operation, the identify operation, the remove operation, and the re-normalize operation until k documents are inserted into the new batch of unlabeled instances B c .

19. The method of claim 16 , wherein the current version of the classification model M is created by:

obtaining an unlabeled set of documents D;

obtaining a batch size k;

constructing a first batch of k documents D;

obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and

constructing the current version of the classification model M using the training data documents.

20. The method of claim 19 , further comprising:

obtaining labels for the new batch of unlabeled instances B c ; and

adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .