Assessing survey presentations
View Patent ↗Measurement errors arising from presentation aspects of a survey are detected and can be corrected, based upon responses from a selected set of respondents to different survey presentations. The survey presentations are different not in their information content, but in their presentation, such as using a different order of questions. An alternative presentation or presentations for the survey can be derived such that the measurement error is reduced. Accordingly, an “unbiased” survey can be designed in the pre-deployment phase, or measurement errors can be detected and corrected during the deployment phase.
1. A computer-implemented method for assessing survey presentations, said method comprising:
generating, by a computer, a plurality of survey presentations, each survey presentation having a same information content and number of questions, and a different order of said number of questions, wherein
an order for each of said number of questions is defined by a distance in a feature space between a question and another question in said order,
each question has a number of features including word phrasing, color, and display type, and each feature of said number of features is defined by a different value of a distance metric in said feature space;
offering, by said computer, said plurality of survey presentations to selections of a population grouped into sets of similar profiles to obtain observed responses in said feature space;
estimating, by said computer, an unbiased response in said feature space from said observed responses, based upon a least mean square estimation of said observed responses to said plurality of survey presentations;
determining, by said computer, a measurement error of the difference between said observed responses in said feature space and said unbiased response in said feature space for each question and for each of said sets of similar profiles; and
removing, by said computer, those survey presentations, exceeding a threshold value of said measurement error, based on said difference of said observed responses in said feature space and said unbiased response in said feature space, from said plurality of survey presentations to provide a subset of favorably assessed survey presentations.
2. The method as claimed in claim 1 , further comprising identifying a survey presentation having a lowest minimum measurement error.
3. The method as claimed in claim 1 , further comprising minimizing an expression for said measurement error that incorporates parametric functions relating to a first position of a first question in each of said plurality of survey presentations and to first and second positions of first and second questions in each of said plurality of survey presentations.
4. The method as claimed in claim 1 , wherein said those survey presentations exceeding a threshold value of said measurement error comprise outliers of said observed responses.
5. The method as claimed in claim 1 , wherein said selections of a population are categorized by at least one of age, sex, country, political affiliation, and income group.
6. A computer program product for assessing survey presentations comprising computer readable program code recorded on a computer-readable storage medium for performing a method comprising:
generating a plurality of survey presentations, each survey presentation having a same information content and number of questions, and a different order of said number of questions, wherein
an order for each of said number of questions is defined by a distance in a feature space between a question and another question in said order,
each question has a number of features including word phrasing, color, and display type, and each feature of said number of features is defined by a different value of a distance metric in said feature space;
offering said plurality of survey presentations to selections of a population grouped into sets of similar profiles to obtain observed responses in said feature space;
estimating an unbiased response in said feature space from said observed responses, based upon a least mean square estimation of said observed responses to said plurality of survey presentations;
determining a measurement error of the difference between said observed responses in said feature space and said unbiased response in said feature space for each question and for each of said sets of similar profiles; and
removing those survey presentations, exceeding a threshold value of said measurement error, based on said difference of said observed responses in said feature space and said unbiased response in said feature space, from said plurality of survey presentations to provide a subset of favorably assessed survey presentations.
7. The computer program product as claimed in claim 6 , wherein said method further comprises identifying a survey presentation having a lowest minimum measurement error.
8. The computer program product as claimed in claim 6 , wherein said method further comprises minimizing an expression for said measurement error that incorporates parametric functions relating to a first position of a first question in each of said plurality of survey presentations and to first and second positions of first and second questions in each of said plurality of survey presentations.
9. The computer program product as claimed in claim 6 , wherein said those survey presentations exceeding a threshold value of said measurement error comprise outliers of said observed responses.
10. The computer program product as claimed in claim 6 , wherein said selections of a population are categorized by at least one of age, sex, country, political affiliation, and income group.
11. A computer-implemented method for assessing survey presentations, said method comprising:
generating, by a computer, a plurality of survey presentations, each survey presentation having a same information content and number of questions, and a different order of said number of questions, wherein
an order for each of said number of questions is defined by a distance in a feature space between a question and another question in said order,
each question has a number of features including word phrasing, color, and display type, and each feature of said number of features is defined by a different value of a distance metric in said feature space;
offering, by said computer, said plurality of survey presentations to selections of a population grouped into sets of similar profiles to obtain observed responses in said feature space;
estimating, by said computer, an unbiased response by using a penalty regularization function that measures entropy of probability distribution functions for said observed responses in said feature space;
computing, by said computer, a measurement error for each of said plurality of survey presentations corresponding to a deviation between said observed responses in said feature space and said estimated unbiased response in said feature space, wherein said deviation is calculated by a KL divergence/JS divergence measure; and
selecting, by said computer, those survey presentations, beneath a threshold value of said measurement error, based on said deviation between said observed responses in said feature space and said estimated unbiased responses in said feature space, from said plurality of survey presentations to provide favorably assessed survey presentations.
12. The method as claimed in claim 11 , further comprising identifying a survey presentation having a lowest minimum measurement error.
13. The method as claimed in claim 11 , wherein said those survey presentations exceeding a threshold value of said measurement error comprise outliers of said observed responses.
14. The method as claimed in claim 11 , wherein said selections of a population are categorized by at least one of age, sex, country, political affiliation, and income group.