System and method for propagating classification decisions
View Patent ↗A system and method for propagating classification decisions is provided. Text marked within one or more unclassified documents that is determined to be responsive to a predetermined issue is received from a user. The unclassified documents are selected from a corpus. A search query is generated from the responsive text. Same result documents are identified by applying inclusive search parameters to the query, applying the search query to the corpus, and identifying the documents that satisfy the query. Similar result documents are identified by adjusting a breadth of the query by applying less inclusive search parameters and identifying documents from the corpus that satisfy the query. A responsive classification code is automatically assigned to each same result document for classification as responsive documents. The similar documents are provided to the user. A responsive classification decision is received from the user for classification as the responsive documents.
1. A system for propagating classification decisions, comprising:
a quantity module to determine a quantity of unclassified documents for review by a user, wherein the quantity of the unclassified documents is determined in accordance with the equation:
n
=
z
2
pq
E
2
where z is an abscissa of the normal curve, E is a desired level of precision, p is a probability that an unclassified document includes responsive text, and q is a probability that the unclassified document fails to include responsive text;
a document feeder to randomly select the n quantity of the unclassified documents from a corpus for providing to the user;
a receipt module to receive from the user, text marked within one or more of the randomly selected n quantity of the unclassified documents that is responsive to a predetermined issue;
a query generator to generate a search query from the responsive text and to identify result documents, comprising:
a same search module to identify same result documents by applying inclusive search parameters to the query, by applying the search query to the corpus, and by identifying the documents that satisfy the query as the same result documents; and
a similar search module to identify similar result documents from the corpus by adjusting a breath of the query by applying less inclusive search parameters and by identifying documents from the corpus that satisfy the query as the similar result documents;
a propagator to automatically assign a responsive classification code to the same result documents for classification as responsive documents;
the document feeder to provide the similar documents to the user and to receive a responsive classification decision from the user for at least one of the similar documents for classification as the responsive documents; and
a processor to execute the modules, document feeder, query generator, and propagator.
2. A system according to claim 1 , further comprising:
a selection module to select further unclassified documents from the corpus based on a similarity of each further unclassified document to the responsive text.
3. A system according to claim 1 , further comprising:
a document calculation module to determine a total expected number of the responsive documents in the corpus and to provide the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.
4. A system according to claim 3 , wherein the total expected number of responsive documents is calculated based on a number of the responsive documents and a total number of documents in the corpus.
5. A system according to claim 1 , wherein the marked text is identified by a marking indicator comprising at least one of a text box, highlighting, underlining, font style, and font size.
6. A system according to claim 1 , wherein the query generator generates at least one further search query based on a further text selection that is responsive to the predetermined issue and identifies further responsive documents as those that satisfy the further search query.
7. A system according to claim 1 , further comprising:
a validator to select a sample of the unclassified documents remaining in the corpus, to analyze the selected unclassified documents for responsive text, and to transmit further unclassified documents from the corpus to the user for review when at least one of the selected unclassified documents includes responsive text.
8. A system according to claim 7 , further comprising:
a document calculation module to calculate a number of the unclassified documents for inclusion in the sample in accordance with the equation:
M
=
x
b
where b is an upper bound value representing a percentage of the unclassified documents remaining in the corpus that are responsive and x is an integer that is determined based on a desired confidence level that all the responsive documents are identified.
9. A system according to claim 1 , further comprising:
a quality control module to edit the search query based on feedback from the user.
10. A method for propagating classification decisions, comprising:
determining a quantity of unclassified documents for review by a user, wherein the quantity of the unclassified documents is determined in accordance with the equation:
n
=
z
2
pq
E
2
where z is an abscissa of the normal curve, E is a desired level of precision, p is a probability that an unclassified document includes responsive text, and q is a probability that the unclassified document fails to include responsive text;
randomly selecting the n quantity of the unclassified documents from a corpus for providing to the user;
receiving from the user, text marked within one or more of the randomly selected n quantity of the unclassified documents that is responsive to a predetermined issue;
generating a search query from the responsive text;
identifying same result documents, comprising:
applying inclusive search parameters to the query;
applying the search query to the corpus; and
identifying the documents that satisfy the query as the same result documents;
identifying similar result documents from the corpus, comprising:
adjusting a breath of the query by applying less inclusive search parameters; and
identifying documents from the corpus that satisfy the query as the similar result documents;
automatically assigning a responsive classification code to the same result documents for classification as responsive documents; and
providing the similar documents to the user and receiving a responsive classification decision from the user for at least one of the similar documents for classification as the responsive documents.
11. A method according to claim 10 , further comprising:
selecting further unclassified documents from the corpus based on a similarity of each of the further unclassified documents to the responsive text.
12. A method according to claim 10 , further comprising:
determining a total expected number of the responsive documents in the corpus; and
providing the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.
13. A method according to claim 12 , wherein the total expected number of responsive documents is calculated based on a a number of the responsive documents and a total number of documents in the corpus.
14. A method according to claim 10 , wherein the marked text is identified by a marking indicator comprising at least one of a text box, highlighting, underlining, font style, and font size.
15. A method according to claim 10 , further comprising:
identifying further responsive documents for the predetermined issue, comprising;
generating at least one further search query based on a further text selection that is responsive to the predetermined issue; and
selecting the further responsive documents as those that satisfy the further search query.
16. A method according to claim 10 , further comprising:
selecting a sample of the unclassified documents remaining in the corpus;
analyzing the selected unclassified documents for responsive text; and
transmitting further unclassified documents from the corpus to the user for review when at least one of the selected unclassified documents includes responsive text.
17. A method according to claim 16 , further comprising:
calculating a number of unclassified documents for inclusion in the sample in accordance with the equation:
M
=
x
b
where b is an upper bound value representing a percentage of the unclassified documents remaining in the corpus that are responsive and x is an integer that is determined based on a desired confidence level that all the responsive documents are identified.
18. A method according to claim 10 , further comprising:
editing the search query based on feedback from the user.
19. A system for propagating classification decisions, comprising:
text marked within one or more unclassified documents that is responsive to a predetermined issue, wherein the one or more unclassified documents are selected from a corpus of unclassified documents;
a query generator to generate a search query from the responsive text and to identify result documents, comprising:
a same search module to identify same result documents by applying inclusive search parameters to the query, by applying the search query to the corpus, and by identifying the documents that satisfy the query as the same result documents; and
a similar search module to identify similar result documents from the corpus by adjusting a breath of the query by applying less inclusive search parameters and by identifying documents from the corpus that satisfy the query as the similar result documents;
a propagator to automatically assign a responsive classification code to the same result documents for classification as responsive documents;
a document feeder to provide the similar documents to the user and to receive a responsive classification decision from the user for at least one of the similar documents for classification as the responsive documents;
a validation module to provide a number of the unclassified documents remaining in the corpus for further review, wherein the number of remaining unclassified documents is determined in accordance with the equation:
M
=
x
b
where b is an upper bound value representing a percentage of the unclassified documents remaining in the corpus that are responsive and x is an integer that is determined based on a desired confidence level that all the responsive documents are identified; and
a processor to execute the modules, the query generator, the propagator, and the document feeder.
20. A system according to claim 19 , further comprising:
a document selection module to randomly select the M remaining unclassified documents for review.
21. A system according to claim 19 , further comprising:
a responsive document determination module to perform one of determining that all the responsive documents in the corpus have been identified when none of the M remaining unclassified documents include responsive material and selecting further remaining unclassified documents for review when at least one of the remaining unclassified documents includes responsive material.
22. A system according to claim 19 , wherein the document feeder further provides the one or more unclassified documents to the user for review via at least one of randomly selecting each of the one or more unclassified documents, selecting each of the one or more unclassified documents based on a similarity of that unclassified document to the responsive text, and randomly selecting a first set of documents from the one or more unclassified documents and selecting the remaining unclassified documents from the one or more unclassified documents based on the similarity of that unclassified document to the responsive text.
23. A system according to claim 19 , further comprising:
a document calculation module to determine a total expected number of the responsive documents in the corpus, wherein the total expected number of responsive documents is calculated based on a number of the responsive documents and a total number of documents in the corpus, and to further provide the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.
24. A method for propagating classification decisions, comprising:
receiving from a user, text marked within one or more unclassified documents that is responsive to a predetermined issue, wherein the one or more unclassified documents are selected from a corpus of unclassified documents;
generating a search query from the responsive text;
identifying same result documents, comprising:
applying inclusive search parameters to the query;
applying the search query to the corpus; and
identifying the documents that satisfy the query as the same result documents;
identifying similar result documents from the corpus, comprising:
adjusting a breath of the query by applying less inclusive search parameters; and
identifying documents from the corpus that satisfy the query as the similar result documents;
automatically assigning a responsive classification code to the same result documents for classification as responsive documents;
providing the similar documents to the user and receiving a responsive classification decision from the user for at least one of the similar documents for classification as the responsive documents; and
providing a number of the unclassified documents remaining in the corpus for further review, wherein the number of remaining unclassified documents for further review is determined in accordance with the equation:
M
=
x
b
where b is an upper bound value representing a percentage of the unclassified documents remaining in the corpus that are responsive and x is an integer that is determined based on a desired confidence level that all the responsive documents are identified.
25. A method according to claim 24 , further comprising:
randomly selecting the M remaining unclassified documents for review.
26. A method according to claim 24 , further comprising at least one of:
determining that all the responsive documents in the corpus have been identified when none of the M remaining unclassified documents include responsive text; and
selecting further remaining unclassified documents for review when at least one of the remaining unclassified documents includes responsive text.
27. A method according to claim 24 , further comprising:
providing the one or more unclassified documents to the user for review, comprising at least one of:
randomly selecting each of the one or more unclassified documents;
selecting each of the one or more unclassified documents based on a similarity of that unclassified document to the responsive text; and
randomly selecting a first set of documents from the one or more unclassified documents and selecting the remaining documents from the one or more unclassified documents based on the similarity of that unclassified document to the responsive text.
28. A method according to claim 24 , further comprising:
determining a total expected number of the responsive documents in the corpus based on a number of the responsive documents and a total number of documents in the corpus; and
providing the unclassified documents to the user for review until the total expected number of responsive documents is satisfied.