ADAPTIVE HEAD-TO-HEAD RANKING TO REDUCE SAMPLE SIZE AND IMPROVE DATA QUALITY
A computer-implemented method of gathering data includes defining a list of items to be ranked, identifying a pivot item within the list, collecting data from users providing head-to-head comparisons between other items in the list to be ranked to the pivot item, producing a greater-than-pivot list and a lesser-than-pivot list as next lists, placing the pivot item in a final position in the list, and using the greater-than-pivot list and lesser-than-pivot list separately as the next lists of items to be ranked, repeating the identifying, collecting and placing until all items in the list are in final positions.
1 . A computer-implemented method of gathering data, comprising:
defining a list of items to be ranked;
identifying a pivot item within the list;
collecting data from users providing head-to-head comparisons between other items in the list to be ranked to the pivot item;
producing a greater-than-pivot list and a lesser-than-pivot list as next lists;
placing the pivot item in a final position in the list; and
using the greater-than-pivot list and lesser-than-pivot list separately as the next lists of items to be ranked, repeating the identifying, collecting and placing until all items in the list are in final positions.
2 . The computer-implemented method as claimed in claim 1 , further comprising randomly ordering the list of items to be ranked prior to identifying the pivot item.
3 . The computer-implemented method as claimed in claim 1 , wherein identifying a pivot item within the list comprises identifying a pivot item using prior knowledge of items on the list.
4 . The computer-implemented method as claimed in claim 1 , wherein identifying the pivot item within the list comprises identifying a pivot item using random selection.
5 . The computer-implemented method as claimed in claim 1 , wherein collecting data from users comprises collecting data from users until a stopping point is reached.
6 . The computer-implemented method as claimed in claim 5 , wherein the stopping point comprises determination of a winner of the comparison.
7 . The computer-implemented method as claimed in claim 5 , wherein the stopping point comprises reaching a desired sample size.
8 . The computer-implemented method as claimed in claim 7 , wherein the desired sample size is based upon a confidence level.
9 . The computer-implemented method as claimed in claim 1 , wherein defining the list of items to be ranked comprises defining multiple lists and collecting data comprises collecting data from multiple comparisons from each user.
10 . The computer-implemented method as claimed in claim 1 , wherein the next lists are pre-ordered prior to identifying a new pivot in each list based upon information gathered during a previous iteration of the process.
11 . The computer-implemented method as claimed in claim 1 , wherein the method returns statistically significant data in a fewer number of responses than a traditional ranking method.