Result aggregation
Methods and systems for result aggregation in crowdsourcing are described. In one implementation, a task with a plurality of solution options is provided for workers to respond. Responses for the task are received from multiple workers, where each of the responses includes a worker response confidence value corresponding to one of the plurality of solution options. The worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct. A task answer for the task is estimated based on at least the worker response confidence value received from each of the multiple workers. A performance bonus for each of the multiple workers is computed based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.
1. A method for result aggregation in a crowdsourcing environment, the method comprising:
providing a task with a plurality of solution options and a range of confidence values corresponding to the plurality of solution options to multiple workers;
receiving responses for the task from the multiple workers, wherein each of the responses comprises at least a worker's selection of one of the plurality of solution options and a worker response confidence value selected from the range of confidence values corresponding to the selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct;
estimating a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
computing a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.
2. The method as claimed in claim 1 , wherein the estimating the task answer is based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task difficulty factor for the task, wherein the task difficulty factor is indicative of a probability of a random worker responding to the task correctly.
3. The method as claimed in claim 2 , wherein the estimating the task answer is based on boundary conditions indicative of biased nature of the multiple workers.
4. The method as claimed in claim 2 , wherein the estimating the task answer is based on boundary conditions indicative of unbiased nature of the multiple workers.
5. The method as claimed in claim 1 , wherein the estimating the task answer is based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task suitability factor for the task, wherein the task suitability factor is indicative of a probability of suitability of the task for a worker.
6. The method as claimed in claim 1 , wherein at least one of the responses further comprises a plurality of worker response confidence values, each of the plurality of worker response confidence values respectively corresponding to a solution option from amongst the plurality of solution options.
7. The method as claimed in claim 1 , wherein the Proper Scoring Rule is one of a logarithmic-based Proper Scoring Rule, a quadratic-based Proper Scoring Rule, and a power-based Proper Scoring Rule.
8. A result aggregation system comprising:
a processor; and
a memory storing instructions that when executed by the processor cause the processor to:
provide a task with a plurality of solution options to multiple workers;
receive responses for the task from the multiple workers, wherein each of the responses comprises at least one worker response confidence value corresponding to at least one of the plurality of solution options, wherein the worker response confidence value is indicative of confidence of a worker for the at least one of the plurality of solution options to be correct and the worker response confidence value is selected from a range of confidence values;
estimate a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
compute a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.
9. The result aggregation system as claimed in claim 8 , wherein the processor is to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task difficulty factor for the task, wherein the task difficulty factor is indicative of a probability of a random worker responding to the task correctly.
10. The result aggregation system as claimed in claim 8 , wherein the processor is to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task suitability factor for the task, wherein the task suitability factor is indicative of a probability of suitability of the task for a worker.
11. The result aggregation system as claimed in claim 10 , wherein the task answer is estimated based on boundary conditions indicative of biased nature of the multiple workers.
12. The result aggregation system as claimed in claim 8 , wherein the Proper Scoring Rule is one of a logarithmic-based Proper Scoring Rule, a quadratic-based Proper Scoring Rule, and a power-based Proper Scoring Rule.
13. A non-transitory computer-readable medium comprising instructions for result aggregation, wherein the instructions are executable by a processor to cause the processor to:
provide a task with a plurality of solution options to multiple workers via a Graphical User Interface (GUI) that includes an input mechanism that receives worker response confidence values corresponding to the plurality of solution options;
receive responses for the task from the multiple workers, wherein each of the responses comprises an input for the worker response confidence value corresponding to a selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct;
estimate a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
compute a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.
14. The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions are to cause the processor to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task difficulty factor for the task, wherein the task difficulty factor is indicative of a probability of a random worker responding to the task correctly.
15. The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions are to cause the processor to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task suitability factor for the task, wherein the task suitability factor is indicative of a probability of suitability of the task for a worker.
16. The non-transitory computer-readable medium as claimed in claim 13 , wherein the input mechanism provided on the GUI comprises slider type inputs.
17. The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions are to cause the processor to:
for each of the responses:
set a system response confidence value for the selected option as equal to the worker response confidence value; and
identify left-over solution options of the plurality of solution options that are not selected in the response.
18. The non-transitory computer-readable medium as claimed in claim 17 , wherein the instructions are to cause the processor to:
obtain a difference between the worker response confidence value and a maxim possible value of the worker response confidence value; and
assign system response confidence values to the left-over solution options by dividing the difference equally between each of the left-over solution options.
19. The non-transitory computer-readable medium as claimed in claim 18 , wherein the instructions to compute a performance bonus for each of the multiple workers are to cause the processor to:
for each worker of the multiple workers:
obtain a worker revealed confidence distribution for the task based on the worker response confidence value and the system response confidence values;
determine a performance score based on the worker revealed confidence distribution, the estimated task answer and the Proper Scoring Rule; and
compute the performance bonus for the worker as a linear transformation of the performance score.
20. The result aggregation system of claim 8 , wherein each of the responses comprises a plurality of worker response confidence values corresponding to the plurality of solution options.