Method and system for assigning a task to be processed by a crowdsourcing platform
View Patent ↗A method and system for applying a task to a crowdsourcing platform for processing are disclosed. For example, a method forwards the task having a question to the crowdsourcing platform, and receives a plurality of answers to the question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform. The method analyzes the plurality of answers on a per-worker basis to determine an adjustment, and applies the adjustment to the plurality of answers.
1. A method for applying a task to a crowdsourcing platform for processing, comprising:
forwarding, with a first computing system, the task having a question to the crowdsourcing platform, the task comprising a human intelligence task associated with an identified compensation per human intelligence task, wherein a correct answer to the question is not pre-identified by the first computing system before the forwarding;
receiving a plurality of answers to the question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform;
analyzing, with the first computing system, the plurality of answers on a per-worker basis to identify one or more indications of accuracy of one or more answers of the plurality of answers and to determine an adjustment at least partially based on the one or more indications of accuracy of one or more answers of the plurality of answers, wherein the analyzing comprises determining one or more relations of one or more answers of the plurality of answers to one or more subsets of answers of the plurality of answers, wherein the adjustment is based at least in part on the one or more relations and an entropy characterization of the plurality of answers; and
applying the adjustment to one or more of the plurality of answers.
2. The method of claim 1 , wherein the adjustment comprises discarding an answer from the plurality of answers for a particular worker of the plurality of workers.
3. The method of claim 1 , wherein the adjustment comprises weighting an answer from the plurality of answers for a particular worker of the plurality of workers.
4. The method of claim 1 , wherein the analyzing is based on a consensus of an answer of a particular worker as compared to answers of other workers of the plurality of workers.
5. The method of claim 1 , further comprising:
modifying a reputation rating of a particular worker of the plurality of workers based on the analyzing.
6. The method of claim 1 , further comprising:
applying a payment adjustment to a particular worker of the plurality of workers based on the analyzing.
7. The method of claim 6 , wherein the payment adjustment comprises an increased payment to be provided to the particular worker.
8. The method of claim 6 , wherein the payment adjustment comprises a decreased payment to be provided to the particular worker.
9. A method for applying a task to a crowdsourcing platform for processing, comprising:
forwarding, with a first computing system, the task having a first question to the crowdsourcing platform, the task comprising a human intelligence task associated with an identified compensation per human intelligence task, wherein a correct answer to the first question is not pre-identified by the first computing system before the forwarding;
receiving a plurality of answers to the first question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform;
analyzing, with the first computing system, the plurality of answers on a per-question basis to identify one or more indications of accuracy of one or more answers of the plurality of answers and to determine an adjustment at least partially based on the one or more indications of accuracy of one or more answers of the plurality of answers, wherein the analyzing comprises determining one or more relations of one or more answers of the plurality of answers to one or more subsets of answers of the plurality of answers, wherein the adjustment is based at least in part on the one or more relations and an entropy characterization of the plurality of answers; and
applying the adjustment to the first question.
10. The method of claim 9 , wherein the analyzing comprises:
calculating an entropy for the first question based upon a distribution of the answers to the first question.
11. The method of claim 10 , wherein the adjustment comprises:
discarding the first question if the entropy is indicative of a high entropy.
12. The method of claim 9 , wherein the adjustment comprises:
modifying a weighting factor associated with the first question.
13. The method of claim 12 , wherein the task comprises a plurality of questions including the first question, wherein the task is for calculating a composite solution based on a plurality of answers to the plurality of questions, and wherein each of the plurality of questions has a respective weighting factor in calculating the composite solution.
14. A method for applying a task to a crowdsourcing platform for processing, comprising:
forwarding, with a first computing system, the task having a first question to the crowdsourcing platform, the task comprising a human intelligence task associated with an identified compensation per human intelligence task, wherein a correct answer to the first question is not pre-identified by the first computing system before the forwarding;
receiving a plurality of answers to the first question, wherein the plurality of answers is provided by a plurality of workers associated with the crowdsourcing platform;
analyzing, with the first computing system, the plurality of answers on a per-worker basis to identify one or more indications of accuracy of one or more answers of the plurality of answers and to determine a first adjustment at least partially based on the one or more indications of accuracy of one or more answers of the plurality of answers;
applying the first adjustment to one or more of the plurality of answers;
analyzing the plurality of answers on a per-question basis to determine a second adjustment; and
applying the second adjustment to the first question;
wherein one or both of the first adjustment and the second adjustment are based at least in part on one or more relations of one or more answers of the plurality of answers to one or more subsets of answers of the plurality of answers and an entropy characterization of the plurality of answers.
15. The method of claim 14 , wherein the first adjustment comprises discarding an answer from the plurality of answers for a particular worker of the plurality of workers.
16. The method of claim 14 , wherein the first adjustment comprises weighting an answer from the plurality of answers for a particular worker of the plurality of workers.
17. The method of claim 14 , wherein the analyzing the plurality of answers on the per-worker basis is based on a consensus of an answer of a particular worker as compared to answers of other workers of the plurality of workers.
18. The method of claim 14 , further comprising:
modifying a reputation rating of a particular worker of the plurality of workers based on the analyzing.
19. The method of claim 14 , further comprising:
applying a payment adjustment to a particular worker of the plurality of workers based on the analyzing the plurality of answers on the per-worker basis.
20. The method of claim 14 , wherein the analyzing the plurality of answers on the per-question basis comprises:
calculating an entropy for the first question based upon a distribution of the answers to the first question.