Methods and systems for determining stopping point
A computer-implemented method includes based on a calculated first estimated error rate, second estimated error rate, first uncertain rank count, second uncertain rank count, and target error rate, displaying a stopping point indication. A computing system includes a processor; and a memory storing instructions that, when executed, cause the computing system to: based on a calculated first estimated error rate, second estimated error rate, first uncertain rank count, second uncertain rank count, and target error rate, display a stopping point indication. A non-transitory computer readable medium includes program instructions that when executed, cause a computer system to: based on a calculated first estimated error rate, second estimated error rate, first uncertain rank count, second uncertain rank count, and target error rate, display a stopping point indication.
1. A computer-implemented method for identifying a stopping point of an active learning process, comprising:
calculating a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and
based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate,
displaying, in a display of a computing device, an indication that the stopping point has been reached.
2. The computer-implemented method of claim 1 , wherein calculating the first estimated error rate includes receiving a coverage review indication from a user.
3. The computer-implemented method of claim 1 , wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of documents.
4. The computer-implemented method of claim 1 , wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of document groups.
5. The computer-implemented method of claim 1 , wherein the target error rate is a configurable constant.
6. The computer-implemented method of claim 1 , wherein calculating the first uncertain rank count and the second uncertain rank count includes comparing uncertain rank counts across a configurable number of previous builds.
7. The computer-implemented method of claim 1 , wherein displaying, in the display of the computing device, the indication that the stopping point has been reached includes
generating a message indicating that the stopping point has been reached; and
transmitting the message via one or both of (i) a push message, and (ii) an email message.
8. A computing system for determining a stopping point of an active learning process, comprising:
one or more processors; and
a memory storing instructions that, when executed, cause the computing system to:
calculate a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and
based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate,
display, in a display of a computing device, an indication that the stopping point has been reached.
9. The computing system of claim 8 , the memory including further instructions that when executed, cause the computing system to:
receive a coverage review indication from a user.
10. The computing system of claim 8 , the memory including further instructions that when executed, cause the computing system to:
determine whether a user has coded a minimum number of documents.
11. The computing system of claim 8 , the memory including further instructions that when executed, cause the computing system to:
determine whether a user has coded a minimum number of document groups.
12. The computing system of claim 8 , wherein the target error rate is a configurable constant.
13. The computing system of claim 8 , the memory including further instructions that when executed, cause the computing system to:
compare uncertain rank counts across a configurable number of previous builds.
14. The computing system of claim 8 , the memory including further instructions that when executed, cause the computing system to:
generate a message indicating that the stopping point has been reached; and
transmit the message via one or both of (i) a push message, and (ii) an email message.
15. A non-transitory computer readable medium storing program instructions that when executed, cause a computer system to:
calculate a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and
based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate,
display, in a display of a computing device, an indication that a stopping point has been reached.
16. The non-transitory computer readable medium of claim 15 , including further program instructions that when executed, cause a computer system to:
receive a coverage review indication from a user.
17. The non-transitory computer readable medium of claim 15 , including further program instructions that when executed, cause a computer system to:
determine whether a user has coded a minimum number of documents.
18. The non-transitory computer readable medium of claim 15 , wherein the target error rate is a configurable constant.
19. The non-transitory computer readable medium of claim 15 , including further program instructions that when executed, cause a computer system to:
compare uncertain rank counts across a configurable number of previous builds.
20. The non-transitory computer readable medium of claim 15 , including further program instructions that when executed, cause a computer system to:
generate a message indicating that the stopping point has been reached; and
transmit the message via one or both of (i) a push message, and (ii) an email message.