A/B testing using sequential hypothesis
A method of executing an A/B test includes, during execution of the A/B test, determining, by a processing device using a sequential frequentist test, that a sample ratio mismatch has occurred, wherein the sample ratio mismatch is determined before the A/B test ends the execution. The method further includes, in response to the determining, ending the execution of the A/B test before a previously scheduled end of the A/B test.
1. A method of executing an A/B test, the method comprising:
during execution of the A/B test, determining, by a processing device using a sequential frequentist test, that a sample ratio mismatch has occurred, wherein the sample ratio mismatch is determined before the A/B test ends the execution;
in response to the determining, ending the execution of the A/B test before a previously scheduled end of the A/B test; and
inverting the sequential frequentist test to determine confidence sequences that possess nominal frequentist coverage probabilities.
2. The method of claim 1 , wherein the sequential frequentist test comprises one or more Bayesian multinomial-Dirichlet families.
3. The method of claim 1 , wherein determining the sample ratio mismatch comprises establishing a Martingale property of one or more posterior odds under a null hypothesis to control a frequentist Type-I error.
4. The method of claim 3 , further comprising at least one of: stopping the A/B test or continuing the A/B test via maximal inequalities based on the determining.
5. The method of claim 1 , further comprising repeating the determining after each observation in the A/B test.
6. The method of claim 5 , wherein the repeating is performed without inflating a frequentist Type-I error associated with the A/B test.
7. A system comprising:
a memory to store data of an A/B test; and
a processing device, operatively coupled to the memory, to:
during execution of the A/B test, determine, using a sequential frequentist test, that a sample ratio mismatch has occurred, wherein the sample ratio mismatch is determined before the A/B test ends the execution;
in response to the determination, end the execution of the A/B test before a previously scheduled end of the A/B test; and
invert the sequential frequentist test to determine confidence sequences that possess nominal frequentist coverage probabilities.
8. The system of claim 7 , wherein the sequential frequentist test comprises one or more Bayesian multinomial-Dirichlet families.
9. The system of claim 7 , wherein, to determine the sample ratio mismatch, the processing device is to establish a Martingale property of one or more posterior odds under a null hypothesis to control a frequentist Type-I error.
10. The system of claim 9 , wherein the processing device is to further perform at least one of: stopping the A/B test or continuing the A/B test via maximal inequalities based on the determining.
11. The system of claim 7 , wherein the processing device is further to repeat the determining after each observation in the A/B test.
12. The system of claim 11 , wherein the processing device is to repeat the determining without inflating a frequentist Type-I error associated with the A/B test.
13. A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:
during execution of an A/B test, determine, by the processing device using a sequential frequentist test, that a sample ratio mismatch has occurred, wherein the sample ratio mismatch is determined before the A/B test ends the execution;
in response to the determination, end the execution of the A/B test before a previously scheduled end of the A/B test; and
invert the sequential frequentist test to determine confidence sequences that possess nominal frequentist coverage probabilities.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the sequential frequentist test comprises one or more Bayesian multinomial-Dirichlet families.
15. The non-transitory computer-readable storage medium of claim 13 , wherein, to determine the sample ratio mismatch, the processing device is to establish a Martingale property of one or more posterior odds under a null hypothesis to control a frequentist Type-I error.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the processing device is to further perform at least one of: stopping the A/B test or continuing the A/B test via maximal inequalities based on the determining.
17. The non-transitory computer-readable storage medium of claim 13 , wherein the processing device is further to repeat the determining after each observation in the A/B test.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the processing device is to repeat the determining without inflating a frequentist Type-I error associated with the A/B test.