A/B testing sample ratio mismatch solver
A method of executing an A/B test includes configuring the A/B test to comprise a first plurality of users in a control group and a second plurality of users in a test group, wherein the first plurality of users and the second plurality of users are to be provided two different versions of a webpage. The method further includes, while the A/B test is executing, determining, by a processing device, that a sample ratio mismatch corresponding to the second plurality of users has occurred, wherein the sample ratio mismatch is determined before the A/B test ends executing. The method further includes, in response to the determining, ending the executing of the A/B test before a previously scheduled end of the A/B test.
1. A method of executing an A/B test, the method comprising:
configuring the A/B test to comprise a first plurality of users in a control group and a second plurality of users in a test group, wherein the first plurality of users and the second plurality of users are to be provided two different versions of a webpage;
while the A/B test is executing, determining, by a processor, that a sample ratio mismatch corresponding to the second plurality of users has occurred by utilizing a sequential frequentist test comprising one or more Bayesian multinomial-Dirichlet families, wherein the sample ratio mismatch is determined before a scheduled completion of the A/B test;
in response to the determining, ending the executing of the A/B test before the scheduled completion of the A/B test; and
inverting the sequential frequentist test to determine confidence sequences that possess frequentist coverage probabilities.
2. The method of claim 1 , wherein the determining comprises detecting a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users.
3. The method of claim 2 , wherein a probability that the sample ratio mismatch has occurred is greater than a predetermined error threshold.
4. The method of claim 1 , wherein the determining occurs before a first one hundred users are provided the A/B test.
5. The method of claim 1 , further comprising sending an error notification identifying the sample ratio mismatch.
6. A system comprising:
a memory to store data of an A/B test; and
a processor, operatively coupled to the memory, to:
configure the A/B test to comprise a first plurality of users in a control group and a second plurality of users in a test group, wherein the first plurality of users and the second plurality of users are to be provided two different versions of a webpage;
during an execution of the A/B test, determine that a sample ratio mismatch corresponding to the second plurality of users has occurred by utilizing a sequential frequentist test comprising one or more Bayesian multinomial-Dirichlet families, wherein the sample ratio mismatch is determined before a scheduled completion of the A/B test;
in response to the determination, end the executing of the A/B test before the scheduled completion of the A/B test; and
invert the sequential frequentist test to determine confidence sequences that possess frequentist coverage probabilities.
7. The system of claim 6 , wherein to determine the sample ratio mismatch the processor is to detect a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users.
8. The system of claim 7 , wherein a probability that the sample ratio mismatch has occurred is greater than a predetermined error threshold.
9. The system of claim 6 , wherein the determination occurs before a first one hundred users are provided the A/B test.
10. The system of claim 6 , the processor further to send an error notification identifying the sample ratio mismatch.
11. A non-transitory computer-readable storage medium including instructions that, when executed by a processor, cause the processor to:
configure an A/B test to comprise a first plurality of users in a control group and a second plurality of users in a test group, wherein the first plurality of users and the second plurality of users are to be provided two different versions of a webpage;
during an execution of the A/B test, determine that a sample ratio mismatch corresponding to the second plurality of users has occurred by utilizing a sequential frequentist test comprising one or more Bayesian multinomial-Dirichlet families, wherein the sample ratio mismatch is determined before a scheduled completion of the A/B test;
in response to the determination, end the executing of the A/B test before the scheduled completion of the A/B test; and
invert the sequential frequentist test to determine confidence sequences that possess frequentist coverage probabilities.
12. The non-transitory computer-readable storage medium of claim 11 , wherein to determine the sample ratio mismatch the processor is to detect a statistically significant departure from a ratio corresponding to the first plurality of users and the second plurality of users.
13. The non-transitory computer-readable storage medium of claim 12 , wherein a probability that the sample ratio mismatch has occurred is greater than a predetermined error threshold.
14. The non-transitory computer-readable storage medium of claim 11 , the processor further to send an error notification identifying the sample ratio mismatch.