Worker answer confidence estimation for worker assessment
Particular embodiments may receive results of a benchmark task performed by a worker, wherein fraud analysis information is provided for each of the results. An accuracy confusion matrix may be determined based on the results. For each of the results, an accuracy confidence score may be determined based on the fraud analysis information. Based on the accuracy confusion matrix and the accuracy confidence score, an accuracy of the worker when performing the benchmark task may be predicted.
1. A method comprising:
receiving, by one or more processors of an information processing system, results of a benchmark task performed by a worker, wherein fraud analysis information is provided for each of the results;
determining, by the one or more processors, an accuracy confusion matrix for the worker based on the received results of the benchmark task;
determining, by the one or more processors, an answer reliability score for each of the results based on the fraud analysis information provided for each of the results;
predicting, based on the accuracy confusion matrix and the answer reliability scores, an accuracy confidence score of a future result from the worker when performing another task; and
when the predicted accuracy confidence score of the future result from the worker fails to satisfy an accuracy quality threshold for the other task, requesting the worker to provide additional results from the benchmark task.
2. The method of claim 1 , further comprising: determining that an accuracy of the worker when performing the benchmark task does not meet an acceptable quality limit for the benchmark task;
and requesting additional results of the benchmark task from at least one additional worker.
3. The method of claim 2 , further comprising: determining a prediction regarding how many of the additional results are required to reach the acceptable quality limit for the benchmark task, wherein a number of additional workers from whom additional results are requested is based on the prediction.
4. The method of claim 1 , wherein the answer reliability scores for the results are determined based on attributes of one or more confirmed instances of fraud assessment.
5. The method of claim 1 , further comprising:
identifying a vector of shared features based on skills needed to accurately perform the benchmark task; and
wherein the accuracy confidence scores are further determined based on the vector of shared features.
6. The method of claim 5 , further comprising:
identifying a vector of shared features for the worker based on skills demonstrated by the worker; and
determining, using a Transfer Learning Independent Bayesian Classifier Combination (TLIBCC) algorithm, optimal values for each of the shared features, wherein the predicted accuracy confidence score of the future result from the worker is further based on the optimal values.
7. One or more computer-readable non-transitory storage media embodying software comprising instructions operable when executed to:
receive, by one or more processors of an information processing system, results of a benchmark task performed by a worker, wherein fraud analysis information is provided for each of the results;
determine, by the one or more processors, an accuracy confusion matrix for the worker based on the received results of the benchmark task;
determine, by the one or more processors, an answer reliability score for each of the results based on the fraud analysis information provided for each of the results;
predict, based on the accuracy confusion matrix and the answer reliability scores, an accuracy confidence score of a future result from the worker when performing another task; and
when the predicted accuracy confidence score of the future result from the worker fails to satisfy an accuracy quality threshold for the other task, request the worker to provide additional results from the benchmark task.
8. The media of claim 7 , the software further comprising instructions operable when executed to: determine that an accuracy of the worker when performing the benchmark task does not meet an acceptable quality limit for the benchmark task;
and request additional results of the benchmark task from at least one additional worker.
9. The media of claim 8 , the software further comprising instructions operable when executed to: determine a prediction regarding how many of the additional results are required to reach the acceptable quality limit for the benchmark task, wherein a number of additional workers from whom additional results are requested is based on the prediction.
10. The media of claim 7 , wherein the answer reliability scores for the results are determined based on attributes of one or more confirmed instances of fraud assessment.
11. The media of claim 7 , the software further comprising instructions operable when executed to:
identify a vector of shared features based on skills needed to accurately perform the benchmark task; and
wherein the accuracy confidence scores are further determined based on the vector of shared features.
12. The media of claim 11 , the software further comprising instructions operable when executed to:
identify a vector of shared features for the worker based on skills demonstrated by the worker; and
determine, using a Transfer Learning Independent Bayesian Classifier Combination (TLIBCC) algorithm, optimal values for each of the shared features, wherein the predicted accuracy confidence score of the future result from the worker is further based on the optimal values.
13. A system comprising one or more processors and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:
receive results of a benchmark task performed by a worker, wherein fraud analysis information is provided for each of the results;
determine, by the one or more processors, an accuracy confusion matrix for the worker based on the received results of the benchmark task;
determine, by the one or more processors, an answer reliability score for each of the results based on the fraud analysis information provided for each of the results;
predict, based on the accuracy confusion matrix and the answer reliability scores, an accuracy confidence score of a future result from the worker when performing another task; and
when the predicted accuracy confidence score of the future result from the worker fails to satisfy an accuracy quality threshold for the other task, request the worker to provide additional results from the benchmark task.
14. The system of claim 13 , the processors being further operable when executing the instructions to: determine that an accuracy of the worker when performing the benchmark task does not meet an acceptable quality limit for the benchmark task;
and request additional results of the benchmark task from at least one additional worker.
15. The system of claim 14 , the processors being further operable when executing the instructions to: determine a prediction regarding how many of the additional results are required to reach the acceptable quality limit for the benchmark task, wherein a number of additional workers from whom additional results are requested is based on the prediction.
16. The system of claim 13 , wherein the answer reliability scores for the results are determined based on attributes of one or more confirmed instances of fraud assessment.
17. The system of claim 13 , the processors being further operable when executing the instructions to:
identify a vector of shared features based on skills needed to accurately perform the benchmark task; and
wherein the accuracy confidence scores are further determined based on the vector of shared features.
18. The system of claim 17 , the processors being further operable when executing the instructions to:
identify a vector of shared features for the worker based on skills demonstrated by the worker; and
determine, using a Transfer Learning Independent Bayesian Classifier Combination (TLIBCC) algorithm, optimal values for each of the shared features, wherein the predicted accuracy confidence score of the future result from the worker is further based on the optimal values.
19. The method of claim 1 , wherein: the benchmark task or other task is one or more actions performed by the worker; the results of the benchmark task or the future result from the other task are an output generated by the worker performing that task;
and wherein the benchmark task and the other task are part of a workflow configured to determine at least one answer to at least one question.
20. The method of claim 1 , wherein determining, by the one or more processor, the answer reliability score for each of the results based on the fraud analysis information provided for each of the results comprises: determining, based on the fraud analysis information that one or more of the results reflects an instance of an incorrect result intentionally submitted for the benchmark task;
an instance where the result is submitted to deceive a task evaluator; an instance where the result is submitted without regard for its accuracy; or an instance where the result is deficient.