Methods, systems, and media for identifying errors in predictive models using annotators
Methods, systems, and media for identifying errors in predictive models using annotators are provided. In some embodiments, a method for evaluating predictive models in classification systems is provided, the method comprising: causing an input region to be presented to a user, where the input region receives an instance from the user that corresponds to a predictive model; retrieving a classification conducted by the predictive model for the received instance and a confidence value associated with the classification; determining whether the received instance has been incorrectly classified by the predictive model; determining a reward associated with the incorrect classification made by the predictive model in response to determining that the received instance has been incorrectly classified by the predictive model, where the reward is based on the confidence value associated with the classification of the received instance; and providing the reward to the user.
1. A method for evaluating predictive models in classification systems, the method comprising:
receiving, using a server that includes a hardware processor, from a user device, an instance of a document, wherein the instance identifies a page having content that may have been incorrectly classified by the predictive model as having a severity level in a particular category of interest;
in response to receiving the instance from the user device, retrieving, using the hardware processor, a classification assigned by a predictive model for the received instance and a confidence value associated with the classification;
determining, using the hardware processor, whether the received instance has been incorrectly classified by the predictive model, wherein the confidence value associated with the classification of the received instance is increased in response to determining that the received instance has been correctly classified by the predictive model;
in response to determining that the received instance has been incorrectly classified by the predictive model, (i) causing, using the hardware processor, the received instance and a corrected classification provided by a user of the user device to be used as training data to update the predictive model, and (ii) determining, using the hardware processor, a reward associated with the incorrect classification made by the predictive model, wherein a value of the reward is proportional to the confidence value associated with the classification of the received instance such that a first value of the reward is designated if the predictive model associated a first confidence value that is greater than a first threshold confidence value with the incorrect classification and a second value of the reward is designated if the predictive model associated a second confidence value that is less than a second threshold confidence value with the incorrect classification; and
causing, using the hardware processor, an indication of the reward to be presented on the user device.
2. The method of claim 1 , further comprising causing an indication of a second reward to be presented on the user device, wherein the second reward is smaller than the reward if the received instance had been determined to be incorrectly classified by the predictive model.
3. The method of claim 1 , further comprising:
determining a number of instances received from the user device over a predetermined time period;
calculating a second reward based at least in part on the number of instances received over the predetermined time period; and
causing an indication of the second reward to be presented on the user device.
4. The method of claim 1 , wherein the instance of the document is received via a user interface presented on the user device, and wherein the user interface indicates a type of content corresponding to the received instance.
5. The method of claim 1 , wherein retrieving the classification assigned by the predictive model comprises transmitting a query indicating information related to the document, wherein the classification is received in response to the query.
6. The method of claim 5 , wherein the information indicated in the query includes links included in the document.
7. The method of claim 1 , wherein the classification is assigned by the predictive model based at least in part on minimizing a cost function associated with misclassification of the received instance.
8. The method of claim 7 , wherein the cost function indicates a first penalty associated with incorrectly classifying the received instance as belonging to a minority class and a second penalty associated with incorrectly classifying the received instance as belonging to a majority class.
9. The method of claim 1 , further comprising selecting a type of predictive model based on a number of training examples available.
10. The method of claim 1 , further comprising selecting a type of predictive model based on a threshold value used to determine the confidence value associated with the classification.
11. A system for evaluating predictive models in classification systems, the system comprising:
a hardware processor that is configured to:
receive, from a user device, an instance of a document, wherein the instance identifies a page having content that may have been incorrectly classified by the predictive model as having a severity level in a particular category of interest;
in response to receiving the instance from the user device, retrieve a classification assigned by a predictive model for the received instance and a confidence value associated with the classification;
determine whether the received instance has been incorrectly classified by the predictive model, wherein the confidence value associated with the classification of the received instance is increased in response to determining that the received instance has been correctly classified by the predictive model;
in response to determining that the received instance has been incorrectly classified by the predictive model, (i) cause the received instance and a corrected classification provided by a user of the user device to be used as training data to update the predictive model, and (ii) determine a reward associated with the incorrect classification made by the predictive model, wherein a value of the reward is proportional to the confidence value associated with the classification of the received instance such that a first value of the reward is designated if the predictive model associated a first confidence value that is greater than a first threshold confidence value with the incorrect classification and a second value of the reward is designated if the predictive model associated a second confidence value that is less than a second threshold confidence value with the incorrect classification; and
cause an indication of the reward to be presented on the user device.
12. The system of claim 11 , wherein the hardware processor is further configured to cause an indication of a second reward to be presented on the user device, wherein the second reward is smaller than the reward if the received instance had been determined to be incorrectly classified by the predictive model.
13. The system of claim 11 , wherein the hardware processor is further configured to:
determine a number of instances received from the user device over a predetermined time period;
calculate a second reward based at least in part on the number of instances received over the predetermined time period; and
cause an indication of the second reward to be presented on the user device.
14. The system of claim 11 , wherein the instance of the document is received via a user interface presented on the user device, and wherein the user interface indicates a type of content corresponding to the received instance.
15. The system of claim 11 , wherein retrieving the classification assigned by the predictive model comprises transmitting a query indicating information related to the document, wherein the classification is received in response to the query.
16. The system of claim 15 , wherein the information indicated in the query includes links included in the document.
17. The system of claim 11 , wherein the classification is assigned by the predictive model based at least in part on minimizing a cost function associated with misclassification of the received instance.
18. The system of claim 17 , wherein the cost function indicates a first penalty associated with incorrectly classifying the received instance as belonging to a minority class and a second penalty associated with incorrectly classifying the received instance as belonging to a majority class.
19. The system of claim 11 , wherein the hardware processor is further configured to select a type of predictive model based on a number of training examples available.
20. The system of claim 11 , wherein the hardware processor is further configured to select a type of predictive model based on a threshold value used to determine the confidence value associated with the classification.