AUTOMATIC RAMP-UP OF CONTROLLED EXPERIMENTS

1 . A method, comprising:

calculating, by one or more computer systems, a first risk associated with ramping up exposure to a first A/B test by a first ramp amount;

using a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test; and

when the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, automatically triggering, by the one or more computer systems, a ramp-up of exposure to the first A/B test by the first ramp amount.

2 . The method of claim 1 , further comprising:

calculating a second risk associated with ramping up exposure to a second A/B test by a second ramp amount;

using a second sequential hypothesis test to compare the second risk with a second risk tolerance for the second A/B test; and

when the second sequential hypothesis test indicates that the second risk exceeds the second risk tolerance, discontinuing ramp-up of exposure to the second A/B test.

3 . The method of claim 1 , further comprising:

calculating a second risk associated with ramping up exposure to a second A/B test by a second ramp amount;

using a second sequential hypothesis test to compare the second risk with a second risk tolerance for the second A/B test; and

when comparison of the second risk with the second risk tolerance by the second sequential hypothesis test is inconclusive at an end of a predefined period, triggering a ramp-up of exposure to the second A/B test by the second ramp amount.

4 . The method of claim 1 , further comprising:

obtaining an initial risk assessment for the first A/B test prior to starting the first A/B test; and

triggering an initial exposure to the first A/B test based on the initial risk assessment.

5 . The method of claim 4 , further comprising:

obtaining the initial exposure and the first ramp amount from a ramp-up plan associated with the initial risk assessment.

6 . The method of claim 4 , wherein the initial risk assessment is obtained from an experimenter associated with the A/B test.

7 . The method of claim 1 , further comprising:

calculating a set of additional risks associated with ramping up exposure to the first A/B test by the first ramp amount;

using the first sequential hypothesis test to compare the additional risks with a set of additional risk tolerances for the first A/B test; and

when the first sequential hypothesis test indicates that a majority of the additional risks is within the corresponding additional risk tolerances and none of the additional risks exceed the corresponding additional risk tolerances, triggering the ramp-up of exposure to the first A/B test by the first ramp amount.

8 . The method of claim 1 , further comprising:

when the ramp-up of exposure to the first A/B test reaches a limit representing a maximum performance assessment for the A/B test, performing additional ramp-up of exposure to a treatment version of the A/B test based on an operational risk associated with the additional ramp-up.

9 . The method of claim 1 , wherein the first risk is calculated using:

a performance metric for the first A/B test;

a proportion of a population affected by the first A/B test; and

the first ramp amount.

10 . The method of claim 1 , wherein ramping up the exposure to the A/B test by the ramp amount comprises:

ramping up the exposure to the first A/B test by a largest ramp amount with a value of the risk that is within the risk tolerance.

11 . The method of claim 1 , wherein the first sequential hypothesis test comprises a generalized sequential probability ratio test.

12 . An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

calculate a first risk associated with ramping up exposure to a first A/B test by a first ramp amount;

use a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test; and

when the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, automatically trigger a ramp-up of exposure to the first A/B test by the first ramp amount.

13 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

calculate a second risk associated with ramping up exposure to a second A/B test by a second ramp amount;

use a second sequential hypothesis test to compare the second risk with a second risk tolerance for the second A/B test; and

when the second sequential hypothesis test indicates that the second risk exceeds the second risk tolerance, discontinue ramp-up of exposure to the second A/B test.

14 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

calculate a second risk associated with ramping up exposure to a second A/B test by a second ramp amount;

use a second sequential hypothesis test to compare the second risk with a second risk tolerance for the second A/B test; and

when comparison of the second risk with the second risk tolerance by the second sequential hypothesis test is inconclusive at an end of a predefined period, trigger a ramp-up of exposure to the second A/B test by the second ramp amount.

15 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

obtain an initial risk assessment for the first A/B test prior to starting the first A/B test;

obtain an initial exposure to the first A/B test and the first ramp amount from a ramp-up plan associated with the initial risk assessment; and

trigger an initial exposure to the first A/B test.

16 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

calculate a set of additional risks associated with ramping up exposure to the first A/B test by the first ramp amount;

use the first sequential hypothesis test to compare the additional risks with a set of additional risk tolerances for the first A/B test; and

when the first sequential hypothesis test indicates that a majority of the additional risks is within the corresponding additional risk tolerances and none of the additional risks exceed the corresponding additional risk tolerances, trigger the ramp-up of exposure to the first A/B test by the first ramp amount.

17 . The apparatus of claim 12 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

when the ramp-up of exposure to the first A/B test reaches a limit representing a maximum performance assessment for the A/B test, perform additional ramp-up of exposure to a treatment version of the A/B test based on an operational risk associated with the additional ramp-up.

18 . The apparatus of claim 12 , wherein the first risk is calculated using:

a performance metric for the first A/B test;

a proportion of a population affected by the first A/B test; and

the first ramp amount.

19 . The apparatus of claim 12 , wherein ramping up the exposure to the A/B test by the ramp amount comprises:

ramping up the exposure to the first A/B test by a largest ramp amount with a value of the risk that is within the risk tolerance.

20 . A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

calculating a first risk associated with ramping up exposure to a first A/B test by a first ramp amount;

using a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test; and

when the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, automatically triggering a ramp-up of exposure to the first A/B test by the first ramp amount.