Method and system for test-duration estimation
View Patent ↗Certain embodiments of the present invention are directed to test-duration estimation. A time needed to run an automated web-page test, or the remaining time needed to complete the automated web-page test, is estimated by estimating a sample size needed to ensure that observed differences in conversion rates are attributable, with a specified statistical significance, to selecting particular factor levels for particular factors that are varied during the automated testing. The estimated sample size is then divided by an average web-page access rate to obtain the remaining testing time to achieve a specified significance level.
1. A method for estimating, by a test-service computer system, a duration of an automated web-page test that is displayed on a computer display device, the method comprising:
finding, by the test-service computer system, a sample size for the web-page test necessary to achieve a pre-specified level of statistical significance by computing a sample size such that estimates of factor-level effects satisfy a specified significance level; and
dividing, by the test-service computer system, the sample size by an average rate of page access.
2. The method of claim 1 wherein finding, by the test-service computer system, the sample size further comprises:
computing a sample size per experiment; and
computing the sample size for the web-page test and the product of the sample size per experiment and the number of experiments.
3. The method of claim 2 wherein computing a sample size per experiment further comprises:
from a list of sample sizes per factor, determining a maximum rounded value from a set of values generated by dividing each sample size per factor by a number of appearances of effect levels.
4. The method of claim 1 further including:
displaying the duration of an automated web-page test within an administrative web page displayed to a test-service-computer-system user prior to selection of the automated web-page test for execution and prior to execution of the automated web-page test; and
displaying the duration of an automated web-page test within a run-time administrative web page displayed to a test-service-computer-system user during execution of the automated web-page test.
5. A testing service comprising:
a computer system; and
a testing-service application that runs on the computer system and that displays, on a computer display screen, a run-time administrative web page that includes information related to a currently executing test and a test-progress feature that displays an indication of a remaining time needed to complete the test, the testing-service application estimating the time needed to complete the test by
finding a sample size for the web-page test necessary to achieve a pre-specified level of statistical significance by computing a sample size such that estimates of factor-level effects satisfy a specified significance level, and
dividing the sample size by an average rate of page access.
6. The testing service of claim 5 wherein the test information further includes
cumulative conversion rates for particular experiments;
factor levels;
factors; and
continuously updated graphs of cumulative page views.
7. The testing service of claim 5 wherein test-progress feature comprises:
a horizontal bar having a left-hand portion that represents a total elapsed time over which the test has already executed and having a right-hand portion that represents a remaining time needed to complete the test according to parameters that specify a desired statistical confidence for the test.
8. The testing service of claim 7 wherein the test-progress feature further comprises:
an indication of a time at which the test began;
an indication of a current time; and
an indication of the estimated time of test completion.
9. The testing service of claim 5 wherein the testing-service application computes the remaining time needed to complete the test by:
modeling observed conversion rates for particular experiments, y i , as the sum of a number of feature-level effects and first-order feature-level-interaction effects
y
i
=
β
0
+
β
1
0
x
1
0
(
i
)
+
β
1
1
x
1
1
(
i
)
+
…
β
1
L
1
x
1
L
1
(
i
)
+
β
2
0
x
2
0
(
i
)
+
β
2
1
x
2
1
(
i
)
+
…
β
2
L
2
x
2
L
2
(
i
)
+
⋮
β
M
0
x
M
0
(
i
)
+
β
M
1
x
M
1
(
i
)
+
…
β
M
L
M
x
M
L
M
(
i
)
+
β
1
,
2
0
,
0
x
1
,
2
0
,
0
(
i
)
+
β
1
,
2
0
,
1
x
1
,
2
0
,
1
(
i
)
+
…
β
1
,
2
L
1
,
L
2
+
β
1
,
3
0
,
0
x
1
,
3
0
,
0
(
i
)
+
β
1
,
3
0
,
1
x
1
,
3
0
,
1
(
i
)
+
…
β
1
,
3
L
1
,
L
3
+
⋮
β
M
-
1
,
M
0
,
0
x
M
-
1
,
M
0
,
0
(
i
)
+
β
M
-
1
,
M
0
,
1
x
M
-
1
,
M
0
,
1
(
i
)
+
…
β
M
-
1
,
M
L
M
-
1
,
L
M
x
M
-
1
,
M
L
M
-
1
,
L
M
(
i
)
where β 0 is a constant effect;
β i j is an effect for level j of factor i;
B i,k j,l is a first-order interaction between level j of factor i and level l of factor k;
M is a number of factors, and
L i is one less than a number of levels for factor i;
providing estimates of the feature-level effects β m and first-order feature-level-interaction effects B l,m p,q for each of the feature levels and first-order feature-level interactions within the test design by analyzing observed conversion rates y i for each of the experiments Y i already run during the course of the test; and
estimating the remaining time as a difference between a computed time that the estimated values of the feature-level effects and first-order feature-level-interaction effects will have all stabilized and will subsequently exhibit lower than a threshold variability and the current time.
10. The testing service of claim 5 wherein the testing-service application computes the remaining time needed to complete the test by:
computationally transforming input parameters from the experiment domain to the factor-levels domain;
computing a sample size to achieve a pre-specified level of statistical significance; and
dividing the sample size by an average rate of page access.
11. The testing service of claim 10 wherein the testing-service application transforms input parameters from the experiment domain to the factor-levels domain by:
applying a statistical model for estimation of factor-level effects expressed as
y i =β 0 +β 1 0 x 1 0 ( i )+β 1 1 x 1 1 ( i )+β 1 2 x 1 2 ( i )+β 2 0 x 2 0 ( i )+β 2 1 x 2 1 ( i )+β 12 00 x 12 00 ( i )+ . . . +β 12 21 x 12 21 ( i )+ε i
where y i is a conversion rate for the i-the experiment, β 0 is an overall mean, β i j , i=1, 2 are factor level effects, β ij l i l j , i=1, 2 are first-order factor-level interactions effects, and ε i is an error that includes a variability in test data and a modeling error, the statistical model alternatively expressed as
ŷ=Xβ
where ŷ is a vector of expectations of y i , X is a matrix of and x k j (i), and β is a vector of β k j effect values; and
estimating the β vector of β k j effect values.
12. The testing service of claim 11 wherein the testing-service application additionally determines model constraints to be satisfied, expressed as:
β 1 0 +β 1 1 +β 1 2 =0
β 2 0 +β 2 1 =0
β 12 00 +β 12 10 +β 12 20 =0
β 12 01 +β 12 11 +β 12 21 =0
β 12 00 +β 12 01 =0
β 12 10 +β 12 11 =0
β 12 20 +β 12 21 =0
which indicate that the different factor-level effects, for a given factor, sum to 0.
13. The testing service of claim 12 wherein the testing-service application determines additional model constraints to be satisfied and incorporates the constraints and statistical model together as:
[
y
^
0
]
=
[
X
H
]
β
where 0 is a vector of zeros of the appropriate dimension and H is a constraint matrix.
14. The testing service of claim 13 wherein the testing-service application estimates β by a least squares process:
{circumflex over (β)}=( {tilde over (X)} T {tilde over (X)} ) −1 {tilde over (X)} T y (Equation 0.7)
where columns of matrix
X
~
=
[
X
H
]
.
15. The testing service of claim 13 wherein the testing-service application estimates β by a least squares process:
β
^
=
N
N
F
(
X
~
T
X
~
)
-
1
(
X
F
)
T
y
where X F is an X matrix that corresponds to a fractional factorial design which includes fewer rows corresponding to fewer experiments, N F is the number of rows in the fractional factorial design, and N is a number of rows in the full factorial design.
16. The testing service of claim 15 wherein the testing-service application computes estimated values of level effects β by one of:
assuming that the conversion rates of all experiments but control and optimal are equal;
assuming that the difference between optimal and control level effects are equal for all factors; and
assuming that conversion rates y i of all experiments other than control and optimal are equal and all control effects β c are equal.
17. The testing service of claim 15 wherein the testing-service application computes a sample size for an experiment by one of:
Dunnett's procedure; and
a stochastic curtailment method.
18. The testing service of claim 10 wherein the testing-service application computes the sample size to achieve a pre-specified level of statistical significance by:
computing a sample size per experiment; and
computing the sample size for the web-page test and the product of the sample size per experiment and the number of experiments.
19. The testing service of claim 12 wherein computing a sample size per experiment further comprises:
from a list of sample sizes per factor, determining a maximum rounded value from a set of values generated by dividing each sample size per factor by a number of appearances of effect levels.