Statistics acceleration in multivariate testing
A method includes determining a plurality of epochs from a timeline of an experiment, wherein each of the plurality of epochs corresponds to a constant traffic allocation. The method further includes calculating, by a processing device, first and second mean values corresponding to a baseline of the experiment and a variation of the experiment for each epoch, respectively. The method further includes determining, for each epoch, a difference between the first mean value and the second mean value to generate a plurality of differences. The method further includes calculating, by the processing device, a weighted sum of the plurality of differences to generate a final difference in mean values.
1. A method comprising:
determining a plurality of epochs from a timeline of an experiment, wherein each of the plurality of epochs corresponds to a time duration comprising a constant traffic allocation between a baseline of the experiment and a variation of the experiment, wherein both the baseline of the experiment and the variation of the experiment are presented to visitors during each epoch of the plurality of epochs according to the respective constant traffic allocation of the time duration corresponding to the epoch, wherein at least two of the plurality of epochs have a different constant traffic allocation, and wherein determining the plurality of epochs is performed in response to detecting time variations over the timeline and a dynamic change to a traffic allocation of the experiment between the baseline of the experiment and the variation of the experiment;
calculating, by a processing device, first and second mean values corresponding to the baseline of the experiment and the variation of the experiment for each epoch, respectively;
determining, for each epoch, a difference between the first mean value and the second mean value to generate a plurality of differences;
calculating, by the processing device, a weighted sum of the plurality of differences to generate a final difference in mean values; and
modifying the experiment based on the final difference in mean values.
2. The method of claim 1 , wherein the constant traffic allocation corresponds to traffic of a website or an application.
3. The method of claim 2 , wherein the website or application is undergoing an A/B test.
4. The method of claim 1 , wherein the weighted sum is weighted in proportion to a number of the visitors in each of the plurality of epochs.
5. The method of claim 1 , further comprising providing the final difference in mean values to a statistical test.
6. The method of claim 5 , wherein the statistical test is a t-test.
7. The method of claim 6 , wherein the statistical test is a sequential test.
8. The method of claim 1 , further comprising:
determining that a threshold number of the visitors was not reached during a first epoch of the plurality of epochs; and
in response to the determining, combining the first epoch with a second epoch.
9. A system comprising:
a memory to store a final difference in mean values;
a processing device, operatively coupled to the memory, to:
determine a plurality of epochs from a timeline of an experiment, wherein each of the plurality of epochs corresponds to a time duration comprising a constant traffic allocation between a baseline of the experiment and a variation of the experiment, wherein both the baseline of the experiment and the variation of the experiment are presented to visitors during each epoch of the plurality of epochs according to the respective constant traffic allocation of the time duration corresponding to the epoch, wherein at least two of the plurality of epochs have a different constant traffic allocation, and wherein determining the plurality of epochs is performed in response to detecting time variations over the timeline and a dynamic change to a traffic allocation of the experiment between the baseline of the experiment and the variation of the experiment;
calculate first and second mean values corresponding to the baseline of the experiment and the variation of the experiment for each epoch, respectively;
determine, for each epoch, a difference between the first mean value and the second mean value to generate a plurality of differences;
calculate a weighted sum of the plurality of differences to generate the final difference in mean values; and
modify the experiment based on the final difference in mean values.
10. The system of claim 9 , wherein the constant traffic allocation corresponds to traffic of a website or an application.
11. The system of claim 10 , wherein the website or application is undergoing an A/B test.
12. The system of claim 9 , wherein each of the plurality of epochs is approximately one hour.
13. The system of claim 9 , wherein the weighted sum is weighted in proportion to a number of the visitors in each of the plurality of epochs.
14. The system of claim 9 , wherein the processing device is further to provide the final difference in mean values to a statistical test.
15. The system of claim 9 , wherein the processing device is further to:
determine that a threshold number of the visitors was not reached during a first epoch of the plurality of epochs; and
in response to the determination, combine the first epoch with a second epoch.
16. A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:
determine a plurality of epochs from a timeline of an experiment, wherein each of the plurality of epochs corresponds to a time duration comprising a constant traffic allocation between a baseline of the experiment and a variation of the experiment, wherein both the baseline of the experiment and the variation of the experiment are presented to visitors during each epoch of the plurality of epochs according to the respective constant traffic allocation of the time duration corresponding to the epoch, wherein at least two of the plurality of epochs have a different constant traffic allocation, and wherein determining the plurality of epochs is performed in response to detecting time variations over the timeline and a dynamic change to a traffic allocation of the experiment between the baseline of the experiment and the variation of the experiment;
calculate, by the processing device, first and second mean values corresponding to the baseline of the experiment and the variation of the experiment for each epoch, respectively;
determine, for each epoch, a difference between the first mean value and the second mean value to generate a plurality of differences;
calculate, by the processing device, a weighted sum of the plurality of differences to generate a final difference in mean values; and
modify the experiment based on the final difference in mean values.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the weighted sum is weighted in proportion to a number of the visitors in each of the plurality of epochs.
18. The non-transitory computer-readable storage medium of claim 16 , providing the final difference in mean values to a statistical test.
19. The non-transitory computer-readable storage medium of claim 16 , further comprising:
determining that a threshold number of the visitors was not reached during a first epoch of the plurality of epochs; and
in response to the determining, combining the first epoch with a second epoch.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the constant traffic allocation corresponds to traffic of a web site or an application.