Sample size determination in sequential hypothesis testing
Sample size determination techniques in sequential hypothesis testing in a digital medium environment are described. The sample size may be determined before a test to define a number of samples (e.g., user interactions with digital marketing content) that are likely to be tested as part of the sequential hypothesis testing in order to achieve a result. The sample size may also be determined in real time to define a number of samples that likely remain for testing in order to achieve a result. The sample size may be determined in a variety of ways, such as through simulation, based on a gap between conversion rates for different options being tested, and so on.
1. In a digital medium testing environment, a method implemented by at least one computing device, the method comprising:
receiving, by the at least one computing device, an input parameter used to define an amount of statistical significance to perform sequential hypothesis testing;
initiating, by the at least one computing device, the sequential hypothesis testing by collecting via a network, in real time, a first set of samples from data describing user interactions with a plurality of items of digital content;
receiving, by the at least one computing device, a change to the input parameter during the collecting of the first set of samples from the data in real time, the change to the input parameter causing a change to the defined amount of statistical significance, the change to the defined amount of statistical significance is caused in real time as the sequential hypothesis testing is performed;
continuing, by the at least one computing device, the sequential hypothesis testing by collecting via the network, in real time, a second set of samples from the data, the second set of samples determined by estimating a number of samples to reach statistical significance based on a difference between a first conversion rate for a first item of the plurality of items and a second conversion rate for a second item of the plurality of items and the change to the defined amount of statistical significance;
completing, by the at least one computing device, the sequential hypothesis testing; and
outputting, by the at least one computing device, a result of the sequential hypothesis testing in a user interface of a display device.
2. The method as described in claim 1 , wherein the defined amount of statistical significance is based at least in part on an amount of permitted error.
3. The method as described in claim 2 , wherein the amount of permitted error references both Type I errors that define false positives and Type II errors that define false negatives.
4. The method as described in claim 1 , wherein the first set of samples are collected using a Bernoulli sampling technique.
5. In a digital medium testing environment, a method implemented by at least one computing device, the method comprising:
receiving, by the at least one computing device, an input parameter used to define statistical significance to perform sequential hypothesis testing, the statistical significance defining a confidence level in accuracy of results of the sequential hypothesis testing;
initiating, by the at least one computing device, the sequential hypothesis testing based on the defined statistical significance by collecting via a network, in real time, increasingly larger number of samples from data describing user interactions with a plurality of items of digital content;
determining, by the at least one computing device, the confidence level in the accuracy of the results during the collecting of the samples in real time;
receiving, by the at least one computing device, a change to the input parameter causing a change to the defined statistical significance, the change to the defined statistical significance caused in real time during the sequential hypothesis testing;
completing, by the at least one computing device, the sequential hypothesis testing by estimating a number of samples using a difference between a first conversion rate for a first item of the plurality of items and a second conversion rate for a second item of the plurality of items and testing the estimated number of samples based on the change to the defined statistical significance; and
outputting, by the at least one computing device, a result of the determining in a user interface in real time during the collecting of the samples as part of the sequential hypothesis testing.
6. The method as described in claim 5 , wherein the statistical significance is based at least in part on an amount of permitted error that references both Type I errors that define false positives and Type II errors that define false negatives.
7. The method as described in claim 1 , wherein the result of the sequential hypothesis testing is a determination whether to reject or not reject a null hypothesis.
8. The method as described in claim 1 , wherein the input parameter includes a false discovery rate.
9. The method as described in claim 1 , further comprising outputting in the user interface, in real time, a percentage of the sequential hypothesis testing completed.
10. The method as described in claim 1 , further comprising outputting in the user interface, in real time, a time to complete the sequential hypothesis testing.
11. The method as described in claim 1 , further comprising outputting in the user interface, in real time, a remaining number of sequential hypothesis tests to be performed.
12. The method as described in claim 5 , further comprising outputting in the user interface, in real time, a percentage of the sequential hypothesis testing completed.
13. The method as described in claim 5 , further comprising outputting in the user interface, in real time, a time to complete the sequential hypothesis testing.
14. The method as described in claim 5 , further comprising outputting in the user interface, in real time, a remaining number of sequential hypothesis tests to be performed.
15. In a digital medium testing environment, a method implemented by at least one computing device, the method comprising:
receiving, by the at least one computing device, an input parameter used to define statistical significance to perform sequential hypothesis testing, the statistical significance defining a confidence level in accuracy of results of the sequential hypothesis testing;
initiating, by the at least one computing device, the sequential hypothesis testing based on the defined statistical significance by collecting via a network, in real time, increasingly larger number of samples from data describing user interactions with a plurality of items of digital content;
receiving, by the at least one computing device, a change to the input parameter causing a change to the defined statistical significance, the change to the defined statistical significance caused in real time during the sequential hypothesis testing;
continuing, by the at least one computing device, the sequential hypothesis testing by estimating a number of samples using a difference between a first conversion rate for a first item of the plurality of items and a second conversion rate for a second item of the plurality of items and testing the estimated number of samples based on the change to the defined statistical significance in real time;
determining, by the at least one computing device, an information output during the collecting of the samples in real time as part of the sequential hypothesis testing; and
outputting, by the at least one computing device, the information output in a user interface in real time during the collecting of the samples as part of the sequential hypothesis testing.
16. The method as described in claim 15 , wherein the information output is a determination whether to reject or not reject a null hypothesis.
17. The method as described in claim 15 , wherein the input parameter includes a false discovery rate.
18. The method as described in claim 15 , further comprising outputting in the user interface, in real time, a percentage of the sequential hypothesis testing completed.
19. The method as described in claim 15 , further comprising outputting in the user interface, in real time, a time to complete the sequential hypothesis testing.
20. The method as described in claim 15 , further comprising outputting in the user interface, in real time, a remaining number of sequential hypothesis tests to be performed.