Adaptive population optimization for amplifying the intelligence of crowds and swarms
System and method amplifying the accuracy of forecasts generated by software systems that harness the collective intelligence of human populations by curating optimized sub-populations through an intelligent selection process. Participants predict event outcomes and/or provide evaluations of their confidence in their predictions. The system determines an outlier score for each participant based on the participant's responses and the relation of the participant's responses to the predictions of the population as a whole. Participants can then be selected from the population based on the participant outlier scores.
1. A system for curating an optimized population of human forecasting participants from a baseline population of human forecasting participants based on an algorithmic analysis of prediction data collected from each participant, the analysis identifying the likelihood that each participant will be a high-performer in a prediction task involving one or more future events, the system comprising:
a processing device including a processor and configured for network communication;
a plurality of application instances wherein each application instance is configured to query a participant, receive input from the queried participant about the prediction task, and be in network communication with the processing device regarding the prediction task, wherein the system is configured to perform the steps of:
query each member of the baseline population of participants about the prediction task comprised of predicting a set of events, wherein each event has a set of possible outcomes including only two possible outcomes;
collect a set of predictions from each participant, each participant interacting with one application instance, wherein each set of predictions includes a predicted outcome for each event of the set of events;
for each event in the set of events, compute one or more support values wherein each support value for each event represents the percentage of participants in the baseline population that predicted a particular outcome within the set of possible outcomes;
for each participant, compute an outlier score for each event, wherein the outlier score is computed by algorithmically corn paring the participant's predicted outcome for that event to the support value for that outcome of that event, wherein the outlier score indicates how well that participant's prediction aligns with the predictions given by the baseline population;
for each participant, determine an outlier index based on the plurality of outlier scores computed for that participant for the set of events, the outlier index indicating how well the set of predictions provided by that participant aligned with the sets of predictions given by the baseline population;
curating an optimized population from the baseline population based at least in part upon a plurality of the outlier indexes, the curation process including at least one selected from the group of (a) culling a plurality of participants from the baseline population in response to the outlier index of each culled participant indicating low alignment compared to other participants, and (b) generating a weighting value for a set of participants in the baseline population, the generated weighting values being lower for participants with an outlier index indicating low alignment as compared to weighting values for participants with an outlier index indicating high alignment;
using curated population information to generate at least one crowd-based or swarm-based prediction for a future event having at least two outcomes; and
comparing the larger support value for each event with a super-majority threshold percentage, wherein a super-majority indicator is assigned to each event where the larger support value exceeds the super-majority indicator.
2. The system for curating the optimized population of participants of claim 1 , further comprising the system configured to perform the step of:
inviting users to participate, wherein the baseline population of participants comprises users who accept the invitation.
3. The system for curating the optimized population of participants of claim 1 , further comprising the system configured to perform the step of:
collecting personal information from the baseline population of participants.
4. The system for curating the optimized population of participants of claim 1 , further comprising the system configured to perform the step of:
after determining the outlier index, selecting a sub-population of participants based at least in part on the outlier index values.
5. The system for curating the optimized population of participants of claim 1 , further comprising the system configured to perform the step of:
collecting, from each participant, a quantitative assessment of the participant's confidence in the participant's predicted outcome for one event.
6. The system for curating the optimized population of participants of claim 5 , the system further configured to perform the step of:
weighting the support values based on at least one confidence assessment.
7. The system for curating the optimized population of participants of claim 1 , further comprising the system configured to perform the step of:
collecting, from each participant, a quantitative assessment of the participant's confidence in their knowledge of a specific knowledge category.
8. A method for curating an optimized population of human forecasting participants from a baseline population of human forecasting participants based on an algorithmic analysis of prediction data collected from each participant, the analysis identifying the likelihood that each participant will be a high-performer in a prediction task involving one or more future events, comprising the steps of:
querying, by a processing device including a processor and configured for networked communication, each member of the baseline population of participants about the prediction task comprised of predicting a set of events, wherein each event has a set of possible outcomes including only two possible outcomes;
collecting, by a plurality of application instances, wherein each application instance receives input from one participant and is in networked communication with the processing device, a set of predictions from each participant, each participant interacting with one application instance, wherein each set of predictions includes a predicted outcome for each event of the set of events;
computing, by the processor for each event in the set of events, one or more support values wherein each support value for each event represents the percentage of participants in the baseline population that predicted a particular outcome within the set of possible outcomes;
computing, by the processor for each participant, an outlier score for each event, wherein the outlier score is computed by algorithmically comparing the participant's predicted outcome for that event to the support value for that outcome of that event, wherein the outlier score indicates how well that participant's prediction aligns with the predictions given by the baseline population; and
determining, by the processor for each participant, an outlier index based on the plurality of outlier scores computed for that participant for the set of events, the outlier index indicating how well the set of predictions provided by that participant aligned with the sets of predictions given by the baseline population;
curating, by the processor, an optimized population from the baseline population based at least in part upon a plurality of the outlier indexes, the curation process including at least one selected from the group of (a) culling a plurality of participants from the baseline population in response to the outlier index of each culled participant indicating low alignment compared to other participants, and (b) generating a weighting value for a set of participants in the baseline population, the generated weighting values being lower for participants with an outlier index indicating low alignment as compared to weighting values for participants with an outlier index indicating high alignment;
using, by the processor, of curated population information to generate at least one crowd-based or swarm-based prediction for a future event having at least two outcomes; and
comparing the larger support value for each event with a super-majority threshold percentage, wherein a super-majority indicator is assigned to each event where the larger support value exceeds the super-majority indicator.
9. The method for curating the optimized population of participants of claim 8 , further comprising the step of:
inviting users to participate, wherein the baseline population of participants comprises users who accept the invitation.
10. The method for curating the optimized population of participants of claim 8 , further comprising the step of:
collecting, by the application instances, personal information from the baseline population of participants.
11. The method for curating the optimized population of participants of claim 8 , further comprising the step of:
selecting, by the processor after determining the outlier index, a sub-population of participants based at least in part on the outlier index values.
12. The method for curating the optimized population of participants of claim 8 , further comprising the step of:
collecting, by the application instances from each participant, a quantitative assessment of the participant's confidence in the participant's predicted outcome for one event.
13. The method for curating the optimized population of participants of claim 12 , further comprising the step of:
weighting the support values based on at least one confidence assessment.
14. The method for curating the optimized population of participants of claim 8 , further comprising the step of:
collecting, by the application instances from each participant, a quantitative assessment of the participant's confidence in their knowledge of a specific knowledge category.