Software version control using forecasts as covariate for experiment variance reduction
Various embodiments can reduce variance in a target metric (e.g., experiment outcome). Embodiments can use historical pre-experiment outcomes to predict a metric (forecasts) that is expected for a future measurement time. The forecasts can then be used as a covariate to reduce the variance of the target metric. When predicting forecasts, various embodiments can use an automation pipeline that can generate better and quicker forecasts. When there are multiple covariates that may be considered to reduce variance in the target metric, various embodiments can use a closed from solution for determining optimal coefficient of each covariate.
1 . A method comprising:
providing a first version of a software application having a first option for a particular functionality of the software application, wherein clients of the software application belong to a plurality of categories, wherein the plurality of categories includes at least 100 categories;
for each category of the plurality of categories, measuring a set of first values of a performance metric of the clients using the first version of the software application, the set of first value measured over a first time period and includes at least 1000 values;
determining a predicted value of the performance metric for each of the plurality of categories of the clients during a second time period, the predicted value determined based on the set of first values of the performance metric during the first time period, and the predicted value accounting for a temporal behavior of the set of first values of the performance metric, wherein the second time period is a current or future time period;
providing a second version of the software application having a second option for the particular functionality of the software application;
measuring a second value of the performance metric for each of the plurality of categories of the clients during the second time period;
for each of the plurality of categories of the clients, determining an auxiliary metric using the second value of the performance metric and the predicted value; and
comparing the second version of the software application to the first version of the software application using the auxiliary metric for each of the plurality of categories of the clients.
2 . The method of claim 1 , wherein the predicted value is determined using a time series model.
3 . The method of claim 2 , wherein the time series model comprises a time series data slicer module, a data preprocessing module, a forecasting module, and a prediction processing module.
4 . The method of claim 3 , wherein the time series data slicer module transforms the set of first values of the performance metric during the first time period into different time units or frequencies to determine transformed historical data.
5 . The method of claim 3 , wherein the data preprocessing module performs adjustments to the set of first values to determine adjusted historical data, and wherein the forecasting module determines the predicted value using the adjusted and transformed historical data.
6 . The method of claim 3 , wherein the prediction processing module performs adjustments on the predicted value, wherein the adjustments include removing effects of outliers.
7 . The method of claim 1 , wherein the predicted value is determined using a regression that best fits the set of first values of the performance metric.
8 . The method of claim 1 , wherein the auxiliary metric is determined by subtracting the predicted value from the second value of the performance metric.
9 . The method of claim 1 , wherein the auxiliary metric is determined by subtracting a value from the second value of the performance metric, wherein the value is determined by subtracting an average of predicted values from the plurality of categories by the predicted value.
10 . The method of claim 1 , further comprising:
determining a control distribution using the auxiliary metric for the plurality of categories provided with the first version of the software application;
determining a treatment distribution using the auxiliary metric for the plurality of categories provided with the second version of the software application; and
comparing the control distribution and the treatment distribution to determine which version of the software application provides better performance.
11 . The method of claim 10 , further comprising:
determining the second version of the software application is better than the first version of the software application; and
deploying the second version of the software application to the clients of the software application.
12 . The method of claim 1 , wherein each category in the plurality of categories is a different service provider for a service within the software application.
13 . A computing device comprising:
one or more processors; and
a computer readable medium coupled to the one or more processors and containing instructions for causing the one or more processors to perform a method comprising:
providing a first version of a software application having a first option for a particular functionality of the software application, wherein clients of the software application belong to a plurality of categories, wherein the plurality of categories includes at least 100 categories;
for each category of the plurality of categories, measuring a set of first values of a performance metric of the clients using the first version of the software application, the set of first value measured over a first time period and includes at least 1000 values;
determining a predicted value of the performance metric for each of the plurality of categories of the clients during a second time period, the predicted value determined based on the set of first values of the performance metric during the first time period, and the predicted value accounting for a temporal behavior of the set of first values of the performance metric, wherein the second time period is a current or future time period;
providing a second version of the software application having a second option for the particular functionality of the software application;
measuring a second value of the performance metric for each of the plurality of categories of the clients during the second time period;
for each of the plurality of categories of the clients, determining an auxiliary metric using the second value of the performance metric and the predicted value; and
comparing the second version of the software application to the first version of the software application using the auxiliary metric for each of the plurality of categories of the clients.
14 . The computing device of claim 13 , wherein the method further comprises:
determining a control distribution using the auxiliary metric for categories provided with the first version of the software application;
determining a treatment distribution using the auxiliary metric for categories provided with the second version of the software application; and
comparing the control distribution and the treatment distribution to determine which version of the software application provides better performance.
15 . The computing device of claim 14 , further comprising:
determining the second version of the software application is better than the first version of the software application; and
deploying the second version of the software application to the clients of the software application.
16 . The computing device of claim 13 , wherein each category in the plurality of categories is a different service provider for a service within the software application.
17 . A non-transitory computer readable medium containing instructions, that when executed, cause one or more processors to perform operations comprising:
providing a first version of a software application having a first option for a particular functionality of the software application, wherein clients of the software application belong to a plurality of categories, wherein the plurality of categories includes at least 100 categories;
for each category of the plurality of categories, measuring a set of first values of a performance metric of the clients using the first version of the software application, the set of first value measured over a first time period and includes at least 1000 values;
determining a predicted value of the performance metric for each of the plurality of categories of the clients during a second time period, the predicted value determined based on the set of first values of the performance metric during the first time period, and the predicted value accounting for a temporal behavior of the set of first values of the performance metric, wherein the second time period is a current or future time period;
providing a second version of the software application having a second option for the particular functionality of the software application;
measuring a second value of the performance metric for each of the plurality of categories of the clients during the second time period;
for each of the plurality of categories of the clients, determining an auxiliary metric using the second value of the performance metric and the predicted value; and
comparing the second version of the software application to the first version of the software application using the auxiliary metric for each of the plurality of categories of the clients.
18 . The computer readable medium of claim 17 , further comprising:
determining a control distribution using the auxiliary metric for the plurality of categories provided with the first version of the software application;
determining a treatment distribution using the auxiliary metric for the plurality of categories provided with the second version of the software application; and
comparing the control distribution and the treatment distribution to determine which version of the software application provides better performance.
19 . The computer readable medium of claim 18 , further comprising:
determining the second version of the software application is better than the first version of the software application; and
deploying the second version of the software application to the clients of the software application.
20 . The computer readable medium of claim 17 , wherein each category in the plurality of categories is a different service provider for a service within the software application.