Automated experiment scheduling

1. An apparatus comprising:

a processor circuit; and

logic to:

commence collection of a specified number of samples of user responses to multiple versions of a user interface in a test of the multiple versions of the user interface, wherein a sample of a user response comprises an action taken with respect to a version of the user interface;

analyze initial ones of the samples for an occurrence of one of a primacy effect or a newness effect, a primacy effect indicated by a statistically significant decrease, at a change in the user interface, in a desired user response from returning users followed by a longer term rise in occurrences of that desired user response as compared to the initial statistically significant decrease, and a newness effect indicated by a statistically significant increase, at a change in the user interface, in a desired user response from returning users followed by a longer term fall in occurrences of the desired user response as compared to that initial significant increase;

discard the initial ones of the samples based on detection of an occurrence of one of a primacy effect or a newness effect;

employ an alternative statistical analysis to analyze the samples which were not discarded as the samples are collected;

determine whether a proposed version of the multiple versions elicits a statistically significant change in a sampled action over a control version of the multiple versions;

terminate the test before the specified number of samples are collected in response to a statistically significant change over the control version elicited by the proposed version; and

complete collection of the specified number of samples and employ a classical statistical analysis to analyze the samples in response to the proposed version not eliciting the statistically significant change.

2. The apparatus of claim 1 , the logic to transmit interaction data conveying different versions of the user interface to each of multiple interaction devices operated by different users to perform the test.

3. The apparatus of claim 1 , the alternative statistical analysis comprising Bayesian analysis and the classical statistical analysis comprising null-hypothesis significance test (NHST).

4. The apparatus of claim 1 , the proposed version comprising a variation of a manner in which content is visually presented in the control version.

5. The apparatus of claim 1 , the logic to:

employ the alternative statistical analysis to determine whether the proposed version is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

cull the proposed version from the test to shorten a duration of the test in response to a determination that the proposed version is statistically unlikely to elicit a statistically significant increase in the sampled action over the control version.

6. The apparatus of claim 1 , the logic to:

employ the alternative statistical analysis to determine whether any proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

terminate the test before the specified number of samples are collected in response to a determination that no proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in the sampled action over the control version.

7. The apparatus of claim 1 , the logic to:

extend a duration of the test to replace the initial ones of the samples with additional samples based on detection of an occurrence of one of a primacy effect or a newness effect.

8. The apparatus of claim 1 , the change in user response comprising at least one of clicking through more advertisements, exploring more webpages of a website, reducing bouncing, downloading more files, or sharing more information.

9. An apparatus comprising:

a processor circuit; and

logic to:

commence collection of a specified number of samples of user responses to multiple versions of a user interface in a test of the multiple versions of the user interface, wherein a sample of a user response comprises an action taken with respect to a version of the user interface;

analyze initial ones of the samples for an occurrence of one of a primacy effect or a newness effect, a primacy effect indicated by a statistically significant decrease, at a change in the user interface, in a desired user response from returning users followed by a longer term rise in occurrences of that desired user response as compared to the initial statistically significant decrease, and a newness effect indicated by a statistically significant increase, at a change in the user interface, in a desired user response from returning users followed by a longer term fall in occurrences of the desired user response as compared to that initial significant increase;

discard the initial ones of the samples based on detection of an occurrence of one of a primacy effect or a newness effect;

employ an alternative statistical analysis to analyze the samples which were not discarded as the samples are collected;

determine whether a proposed version of the multiple versions is statistically likely to elicit a statistically significant change in a sampled action over a control version of the multiple versions;

cull the proposed version from the test to shorten a duration of the test in response to a determination that the proposed version is statistically unlikely to elicit a statistically significant change in user response over the control version; and

complete collection of the specified number of samples and employ a classical statistical analysis to analyze the samples in response to the proposed version not eliciting the statistically significant change.

10. The apparatus of claim 9 , the logic to transmit interaction data conveying different versions of the user interface to each of multiple interaction devices operated by different users to perform the test.

11. The apparatus of claim 9 , the alternative statistical analysis comprising Bayesian analysis and the classical statistical analysis comprising null-hypothesis significance test (NHST).

12. The apparatus of claim 9 , the logic to:

employ the alternative statistical analysis to determine whether the proposed version elicits a statistically significant increase in a sampled action over the control version; and

terminate the test before the specified number of samples are collected in response to a statistically significant increase in the sampled action over the control version elicited by the proposed version.

13. The apparatus of claim 9 , the logic to:

employ the alternative statistical analysis to determine whether any proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

terminate the test before the specified number of samples are collected in response to a determination that no proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in the sampled action over the control version.

14. The apparatus of claim 9 , the logic to:

extend a duration of the test to replace the initial ones of the samples with additional samples based on detection of an occurrence of one of a primacy effect or a newness effect.

15. The apparatus of claim 9 , the logic to:

compare results of the classical statistical analysis to results of the alternative statistical analysis; and

determine whether a model of user behavior employed as a prior data input to the alternative analysis is valid.

16. A computer-implemented method comprising:

transmitting interaction data conveying different versions of a user interface to each of multiple interaction devices operated by different users to perform a test of the different versions of the user interface;

commencing collection of a specified number of samples of user responses to the different versions of the user interface, wherein a sample of a user response comprises an action taken with respect to a version of the user interface;

analyzing initial ones of the samples for an occurrence of one of a primacy effect or a newness effect, a primacy effect indicated by a statistically significant decrease, at a change in the user interface, in a desired user response from returning users followed by a longer term rise in occurrences of that desired user response as compared to the initial statistically significant decrease, and a newness effect indicated by a statistically significant increase, at a change in the user interface, in a desired user response from returning users followed by a longer term fall in occurrences of the desired user response as compared to that initial significant increase;

discarding the initially collected ones of the samples based on detecting an occurrence of one of a primacy effect or a newness effect;

analyzing the samples which were not discarded as the samples are collected using Bayesian analysis;

determining whether a proposed version of the multiple versions elicits a statistically significant change in user response over a control version of the multiple versions;

terminating the test before the specified number of samples are collected in response to the proposed version eliciting a statistically significant change over the control version; and

completing collection of the specified number of samples and analyzing the samples using null-hypothesis significance test (NHST) in response to the proposed version not eliciting the statistically significant change.

17. The computer-implemented method of claim 16 , the proposed version comprising a variation of a manner in which content is visually presented in the control version.

18. The computer-implemented method of claim 16 , the method comprising:

employing Bayesian analysis to determine whether the proposed version is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

culling the proposed version from the test to shorten a duration of the test in response to determining that the proposed version is statistically unlikely to elicit a statistically significant increase in the sampled action over the control version.

19. The computer-implemented method of claim 16 , the method comprising:

employing Bayesian analysis to determine whether any proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

terminating the test before the specified number of samples are collected in response to determining that no proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in the sampled action over the control version.

20. The computer-implemented method of claim 16 , the method comprising:

extending a duration of the test to replace the initial ones of the samples with additional samples based on detection of an occurrence of one of a primacy effect or a newness effect.

21. The computer-implemented method of claim 16 , the change in user response comprising at least one of clicking through more advertisements, exploring more webpages of a website, reducing bouncing, downloading more files, and sharing more information.

22. The computer-implemented method of claim 16 , the method comprising:

comparing results of the NHST analysis to results of the Bayesian analysis; and

determining whether a model of user behavior employed as a prior data input to the Bayesian analysis is valid.

23. At least one machine-readable storage medium comprising instructions that when executed by a computing device, cause the computing device to:

transmit interaction data conveying different versions of a user interface to each of multiple interaction devices operated by different users to perform a test of the different versions of the user interface;

commence collection of a specified number of samples of user responses to the different versions of the user interface, wherein a sample of a user response comprises an action taken with respect to a version of the user interface;

analyze initial ones of the samples for an occurrence of one of a primacy effect or a newness effect, a primacy effect indicated by a statistically significant decrease, at a change in the user interface, in a desired user response from returning users followed by a longer term rise in occurrences of that desired user response as compared to the initial statistically significant decrease, and a newness effect indicated by a statistically significant increase, at a change in the user interface, in a desired user response from returning users followed by a longer term fall in occurrences of the desired user response as compared to that initial significant increase;

discard the initially collected ones of the samples based on detecting an occurrence of one of a primacy effect or a newness effect;

analyze the samples which were not discarded as the samples are collected using Bayesian analysis;

determine whether a proposed version of the multiple versions elicits a statistically significant change in user response over a control version of the multiple versions;

terminate the test before the specified number of samples are collected in response to the proposed version eliciting a statistically significant change over the control version; and

complete collection of the specified number of samples and analyze the samples using null-hypothesis significance test (NHST) in response to the proposed version not eliciting the statistically significant change.

24. The at least one machine-readable storage medium of claim 23 , the computing device caused to:

employ Bayesian analysis to determine whether the proposed version is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

cull the proposed version from the test to shorten a duration of the test in response to determining that the proposed version is statistically unlikely to elicit a statistically significant increase in the sampled action over the control version.

25. The at least one machine-readable storage medium of claim 23 , the computing device caused to:

employ Bayesian analysis to determine whether any proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in a sampled action over the control version; and

terminate the test before the specified number of samples are collected in response to determining that no proposed version of the multiple versions is statistically likely to elicit a statistically significant increase in the sampled action over the control version.

26. The at least one machine-readable storage medium of claim 23 , the computing device caused to:

extend a duration of the test to replace the initial ones of the samples with additional samples based on detection of an occurrence of one of a primacy effect or a newness effect.

27. The at least one machine-readable storage medium of claim 23 , the computing device caused to:

compare results of the NHST analysis to results of the Bayesian analysis; and

determine whether a model of user behavior employed as a prior data input to the Bayesian analysis is valid.