Apparatus and Method of Implementing Batch-Mode Active Learning for Technology-Assisted Review of Documents
The present disclosure relates to the electronic document review field and, more particularly, to various apparatuses and methods of implementing batch-mode active learning for technology-assisted review (TAR) of documents (e.g., legal documents).
1 - 14 . (canceled)
15 . A method to implement a diversity sampler process to select new batches of unlabeled instances, comprising:
by one or more computing devices:
inserting an identified document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D into a new batch of unlabeled instances B c ;
removing documents that have a cosine angle≥t with respect to the inserted document from an unlabeled set of available documents D; and
repeating the insert operation and the remove operation until a new batch of unlabeled instances B c are selected.
16 . The method of claim 15 , wherein the classification model M is a hyperplane.
17 . The method of claim 15 , wherein the unlabeled set of available documents D are sorted in increasing order.
18 . The method of claim 15 , wherein the unlabeled set of available documents D has a batch size k.
19 . The method of claim 18 , wherein the one or more computing devices performs the insert operation and the remove operation until k documents are inserted into the new batch of unlabeled instances B c .
20 . The method of claim 15 , wherein the diversity sampler process is implemented using a support vector machine (SVM).
21 . The method of claim 15 , wherein the diversity sampler process is implemented in a technology-assisted document review.
22 . The method of claim 15 , wherein the current version of the classification model M is created by:
obtaining an unlabeled set of documents D;
obtaining a batch size k;
constructing a first batch of k documents D;
obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and
constructing the current version of the classification model M using the training documents.
23 . The method of claim 22 , further comprising:
obtaining labels for the new batch of unlabeled instances B c ; and
adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .
24 . The method of claim 23 , further comprising constructing an updated classification model M using the extended training data documents D c .
25 . The method of claim 15 , wherein identified the document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D comprises:
obtaining the current version of a classification model M, the unlabeled set of available documents D, and the cosine similarity threshold t;
sorting the unlabeled set of available documents D based on each of the documents absolute distance from the current version of the classification model M to obtain sorted indices I for each document of the unlabeled set of available documents D; and
identifying a document having a nearest sorted index I[1] from the current version of the classification model M.
26 . The method of claim 15 , further comprising obtaining sorted indices I of the sorted unlabeled set of available documents D that have a cosine angle≥t with respect to the inserted document I[1].
27 . The method of claim 26 , further comprising removing the documents with the obtained sorted indices I from the sorted unlabeled set of available documents D.
28 . A system configured to implement a diversity sampler process to select new batches of unlabeled instances, the apparatus comprising:
a processor; and
a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions, whereby the system is operable to:
insert an identified document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D into a new batch of unlabeled instances B c ;
remove documents that have a cosine angle≥t with respect to the inserted document from an unlabeled set of available documents D; and
repeat the insert operation and the remove operation until a new batch of unlabeled instances B c are selected.
29 . The system of claim 28 , wherein the diversity sampler process is implemented using a support vector machine (SVM).
30 . The system of claim 28 , wherein the current version of the classification model M is created by:
obtaining an unlabeled set of documents D;
obtaining a batch size k;
constructing a first batch of k documents D;
obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and
constructing the current version of the classification model M using the training documents.
31 . The system of claim 30 , wherein the system is further operable to:
obtain labels for the new batch of unlabeled instances B c ; and
add the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .
32 . The system of claim 31 , wherein the system is further operable to construct an updated classification model M using the extended training data documents D c .
33 . A method to implement a biased probabilistic sampler process to select new batches of unlabeled instances, comprising:
by one or more computing devices:
inserting an identified document that has a nearest absolute distance from a current version of a classification model M out of an unlabeled set of available documents D into a new batch of unlabeled instances B c ;
removing documents that have a cosine angle≥t with respect to the inserted document from an unlabeled set of available documents D; and
repeating the insert operation and the remove operation until a new batch of unlabeled instances B c are selected.
34 . The method of claim 33 , wherein the unlabeled set of available documents D has a batch size k.
35 . The method of claim 34 , wherein the one or more computing devices performs the insert operation and the remove operation until k documents are inserted into the new batch of unlabeled instances B c .
36 . The method of claim 33 , wherein the current version of the classification model M is created by:
obtaining an unlabeled set of documents D;
obtaining a batch size k;
constructing a first batch of k documents D;
obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents; and
constructing the current version of the classification model M using the training data documents.
37 . The method of claim 36 , further comprising:
obtaining labels for the new batch of unlabeled instances B c ; and
adding the labeled new batch of instances B c to a current version of the training data documents referred to as extended training data documents D c .