Methods and systems for generating a combined metric parameter for A/B testing

1. A computer-implemented method of generating a combined metric parameter, the combined metric parameter usable for executing computer-implemented A/B testing of a computer-implemented service, the computer-implemented service having a control version and a treatment version, the method executed on a server, the method comprising:

receiving, by the server, a set of data, the set of data including a first plurality of feature vectors and a second plurality of feature vectors, the first plurality of feature vectors being associated with user interactions of a first plurality of users with the control version of the computer-implemented service during a first time period, and the second plurality of feature vectors being associated with user interactions of a second plurality of users with the treatment version of the computer-implemented service during the first time period;

acquiring, by the server, a first control metric parameter for the first plurality of feature vectors and a first treatment metric parameter for the second plurality of feature vectors, a combination of the first control metric parameter and the first treatment metric parameter being at least partially indicative of a direction of a change in user interactions between the control version and the treatment version of the computer-implemented service, the direction being one of a negative direction, indicating that the treatment version negatively impacts user experience, and a positive direction, indicating that the treatment version positively impacts user experience;

acquiring, by the server, a second control metric parameter for the first plurality of feature vectors and a second treatment metric parameter for the second plurality of feature vectors, a combination of the second control metric parameter and the second treatment metric parameter being at least partially indicative of a magnitude of the change in the user interactions between the control version and the treatment version of the computer-implemented service;

generating, by the server, a combined control metric parameter and a combined treatment metric parameter, the generating comprising:

computing by the server, a first weight parameter weighing respectively the first control metric parameter and the first treatment metric parameter and a second weight parameter weighing respectively the second control metric parameter and the second treatment metric parameter;

selecting, by the server, a minimal first weight parameter and a maximal second weight parameter;

the combined control metric parameter including the first control metric parameter being weighted by the first weight parameter and the second control metric parameter being weighted by the second weight parameter,

the combined treatment metric parameter including the first treatment metric parameter being weighted by the first weight parameter and the second treatment metric parameter being weighted by the second weight parameter,

the combination of the combined control metric parameter and the combined treatment metric parameter being simultaneously at least partially indicative of the magnitude and the direction of the change in user interactions between the control version and the treatment version of the computer-implemented service during the first time period.

2. The method of claim 1 , further comprising: applying, by the server, the combined control metric parameter and the combined treatment metric parameter on a new set of data to determine the magnitude and the direction of the change in the user interactions, the new set of data comprising a third plurality of vectors associated with user interactions on a control version of the computer-implemented service during a second time period and a fourth plurality of vectors being associated with user interactions on a treatment version of the computer-implemented service during a second time period.

3. The method of claim 1 , wherein:

the first weight parameter is indicative of a relative contribution of the first control metric parameter in the combined control metric parameter and the first treatment metric parameter in the combined treatment metric parameter; and

the second weight parameter is indicative of a relative contribution of the second control metric parameter in the combined control metric parameter and the second treatment metric parameter in the combined treatment metric parameter.

4. The method of claim 3 , wherein the magnitude and the direction of the change in the user interactions between the control version and the treatment version of the computer-implemented service during the first time period has a known magnitude and a known direction.

5. The method of claim 4 , wherein the computing the first weight parameter and the second weight parameter further comprises computing, by the server, for each unit of time in the first time period, a sign of a difference between the first treatment metric parameter and the first control treatment parameter, the sign of the difference being indicative of the direction of the change in the user interactions between the control version and the treatment version of the computer-implemented service, the direction being one of the negative direction and the positive direction, the sign of the difference being defined by:

Δ i =sign( M 1 Bi −M 1 Ai )

where M 1 Ai is the first control metric parameter and M 1 Bi is the first treatment metric parameter.

6. The method of claim 5 , further comprising: computing, by the server, for the each unit of the first time period, a weighted z-score based on the combined control metric parameter and the combined treatment metric parameter, the weighted z-score being indicative of a confidence level in the magnitude of the change in the user interactions between the control version and the treatment version of the computer-implemented service, the weighted z-score being defined by:

Z

A

⁢

B

i

⁡

(

c

1

,

c

2

)

=

c

T

·

M

C

⁢

A

i

-

c

T

·

M

C

⁢

B

i

c

T

·

COV

A

i

·

c

+

c

T

·

COV

B

i

·

c

where

c is a weight vector including the first weight parameter c 1 and the second weight parameter c 2 ;

M CA is the combined control metric parameter and M CB is the combined treatment metric parameter; and

COV A i is a covariance matrix between the first control metric parameter and the second control metric parameter and COV B i is the covariance matrix between the first treatment metric parameter and the second treatment metric parameter.

7. The method of claim 6 , further comprising summing, for the first time period, the sign of the difference between the first treatment metric parameter and the first control treatment parameter multiplied by the weighted z-score for each unit of time in the first time period.

8. The method of claim 7 , wherein the computing, by the server, the first weight parameter and the second weight parameter further comprises solving:

∑

i

=

1

N

⁢

Z

A

⁢

B

i

⁡

(

c

1

,

⁢

c

2

)

·

sign

⁡

(

M

1

B

⁢

i

-

M

1

A

⁢

i

)

=

avg

⁡

(

M

C

⁢

B

)

-

avg

⁡

(

M

C

⁢

A

)

var

⁡

[

avg

⁡

(

M

C

⁢

B

)

-

avg

⁡

(

M

C

⁢

A

)

]

.

9. The method of claim 8 , wherein the solving is performed by using one of a Broyden-Fletcher-Goldfard-Shanno (BFGS) algorithm, a gradient based algorithm or a heuristic algorithm.

10. The method of claim 9 , wherein a sum of the first weight parameter and the second weight parameter is a constant.

11. The method of claim 10 , wherein the selecting, by the server, the minimal first weight parameter and the maximal second weight parameter comprises selecting an optimal value for the minimal first weight parameter and selecting the maximal second weight parameter by subtracting the first weight parameter from the constant.

12. The method of claim 11 , wherein one of the first control metric parameter and the first treatment metric parameter is a daily active users (DAU) measure and one of the second control metric parameter and the second treatment metric parameter is a click-through rate (CTR) measure.

13. The method of claim 1 , wherein:

the first plurality of users is at least a portion of users having interacted with the control version of the computer-implemented service during the first time period.

14. The method of claim 1 , wherein:

the second plurality of users is at least a portion of all users having interacted with the treatment version of the computer-implemented service during the first time period.

15. A system for generating a combined metric parameter, the combined metric parameter usable for executing computer-implemented A/B testing of a computer-implemented service, the computer-implemented service having a control version and a treatment version, the system comprising:

a processor; and

a non-transitory computer-readable medium comprising instructions,

the processor, upon executing the instructions, being configured to cause:

receiving a set of data, the set of data including a first plurality of feature vectors and a second plurality of feature vectors, the first plurality of feature vectors being associated with user interactions of a first plurality of users with the control version of the computer-implemented service during a first time period, and the second plurality of feature vectors being associated with user interactions of a second plurality of users with the treatment version of the computer-implemented service during the first time period;

acquiring a first control metric parameter for the first plurality of feature vectors and a first treatment metric parameter for the second plurality of feature vectors, a combination of the first control metric parameter and the first treatment metric parameter being at least partially indicative of a direction of a change in user interactions between the control version and the treatment version of the computer-implemented service, the direction being one of a negative direction indicating that the treatment version negatively impacts user experience, and a positive direction, indicating that the treatment version positively impacts user experience;

acquiring a second control metric parameter for the first plurality of feature vectors and a second treatment metric parameter for the second plurality of feature vectors, a combination of the second control metric parameter and the second treatment metric parameter being at least partially indicative of a magnitude of the change in the user interactions between the control version and the treatment version of the computer-implemented service;

generating a combined control metric parameter and a combined treatment metric parameter, the generating comprising:

computing a first weight parameter weighing respectively the first control metric parameter and the first treatment metric parameter and a second weight parameter weighing respectively the second control metric parameter and the second treatment metric parameter;

selecting a minimal first weight parameter and a maximal second weight parameter;

the combined control metric parameter including the first control metric parameter being weighted by the first weight parameter and the second control metric parameter being weighted by the second weight parameter,

the combined treatment metric parameter including the first treatment metric parameter being weighted by the first weight parameter and the second treatment metric parameter being weighted by the second weight parameter,

the combination of the combined control metric parameter and the combined treatment metric parameter being simultaneously at least partially indicative of the magnitude and the direction of the change in user interactions between the control version and the treatment version of the computer-implemented service during the first time period.

16. The system of claim 15 , wherein the processor is further configured, upon the execution of the instructions, to cause:

applying the combined control metric parameter and the combined treatment metric parameter on a new set of data to determine the magnitude and the direction of the change in the user interactions, the new set of data comprising a third plurality of vectors associated with user interactions on a control version of the computer-implemented service during a second time period and a fourth plurality of vectors being associated with user interactions on a treatment version of the computer-implemented service during a second time period.

17. The system of claim 16 , wherein:

the first weight parameter is indicative of a relative contribution of the first control metric parameter in the combined control metric parameter and the first treatment metric parameter in the combined treatment metric parameter; and

the second weight parameter is indicative of a relative contribution of the second control metric parameter in the combined control metric parameter and the second treatment metric parameter in the combined treatment metric parameter.

18. The system of claim 17 , wherein:

the magnitude and the direction of the change in the user interactions between the control version and the treatment version of the computer-implemented service during the first time period has a known magnitude and a known direction; and wherein:

the computing the first weight parameter and the second weight parameter further comprises computing, by the server, for each unit of time in the first time period, a sign of a difference between the first treatment metric parameter and the first control treatment parameter, the sign of the difference being indicative of the direction of the change in the user interactions between the control version and the treatment version of the computer-implemented service, the direction being one of the negative direction and the positive direction, the sign of the difference being defined by:

Δ i =sign( M 1 Bi −M 1 Ai )

where M 1 Ai is the first control metric parameter and M 1 Bi is the first treatment metric parameter.

19. The system of claim 18 , wherein the processor is further configured, upon the execution of the instructions, to cause:

computing for the each unit of the first time period, a weighted z-score based on the combined control metric parameter and the combined treatment metric parameter, the weighted z-score being indicative of a confidence level in the magnitude of the change in the user interactions between the control version and the treatment version of the computer-implemented service, the weighted z-score being defined by:

Z

A

⁢

B

i

⁡

(

c

1

,

c

2

)

=

c

T

·

M

C

⁢

A

i

-

c

T

·

M

C

⁢

B

i

c

T

·

COV

A

i

·

c

+

c

T

·

COV

B

i

·

c

,

where:

c is a weight vector including the first weight parameter c 1 and the second weight parameter c 2 ;

M CA is the combined control metric parameter and M CB is the combined treatment metric parameter; and

COV A i is a covariance matrix between the first control metric parameter and the second control metric parameter and COV B i is the covariance matrix between the first treatment metric parameter and the second treatment metric parameter.

20. The system of claim 19 , wherein the processor is further configured, upon the execution of the instructions, to cause:

summing, for the first time period, the sign of the difference between the first treatment metric parameter and the first control treatment parameter multiplied by the weighted z-score for each unit of time in the first time period; and

wherein:

wherein the computing the first weight parameter and the second weight parameter further comprises solving:

∑

i

=

1

N

⁢

Z

A

⁢

B

i

⁡

(

c

1

,

⁢

c

2

)

·

sign

⁡

(

M

1

B

⁢

i

-

M

1

A

⁢

i

)

=

avg

⁡

(

M

C

⁢

B

)

-

avg

⁡

(

M

C

⁢

A

)

var

⁡

[

avg

⁡

(

M

C

⁢

B

)

-

avg

⁡

(

M

C

⁢

A

)

]

.