Method of and system for conducting a controlled experiment using prediction of future user behavior

1. A method of conducting a controlled experiment using prediction of future user behavior, the method executable on at least one hardware server, the method comprising:

collecting behavior data on a first set of users and a second set of users over a first period of time, wherein:

the first set of users is exposed to a control variant of a service;

the second set of users is exposed to a treatment variant of the service; and

the behavior data relates to a performance parameter of the controlled experiment;

based on at least one prediction model applied to the behavior data, calculating predicted values of the performance parameter for each user of the first set of users and each user of the second set of users over a second period of time;

obtaining actual values of the performance parameter for each user of the first set of users and each user of the second set of users based on the behavior data collected during the first period of time;

combining each actual value of the performance parameter for each user of the first set of users with the corresponding predicted value of the performance parameter for each user of the first set of users into a combined value of the performance parameter for each user of the first set of users;

combining each actual value of the performance parameter for each user of the second set of users with the corresponding predicted value of the performance parameter for each user of the second set of users into a combined value of the performance parameter for each user of the second set of users;

calculating an average value of the combined values of the performance parameter for each user of the first set of users;

calculating an average value of the combined values of the performance parameter for each user of the second set of users; and

determining if a statistically significant difference exists between the average value of the combined values of the performance parameter for each user of the first set of users and the average value of the combined values of the performance parameter for each user of the second set of users;

selecting, based on the statistically significant difference, one of the control variant of the service and the treatment variant of the service.

2. The method of claim 1 , wherein the service is an online service.

3. The method of claim 2 , wherein the controlled experiment evaluates a change in execution of the online service.

4. The method of claim 3 , wherein the online service is a search engine.

5. The method of claim 4 , wherein the change in execution is at least one of: a change in a ranking algorithm of the search engine, a change in engine response time of the search engine and a change in a user interface of the search engine.

6. The method of claim 4 , wherein the behavior data comprises measures of user interactions with the search engine.

7. The method of claim 6 , wherein the performance parameter comprises a pre-selected type of user interaction with the search engine.

8. The method of claim 7 , wherein the pre-selected type of user interaction comprises at least one of: a number of sessions per user, a number of queries per user, a number of clicks per user, a presence time of a user, a number of clicks per query of a user and an absence time per session of a user.

9. The method of claim 1 , wherein the first period of time corresponds to a duration of the controlled experiment.

10. The method of claim 1 , wherein the first period of time corresponds to a period of time less than a duration of the controlled experiment.

11. The method of claim 10 , wherein the controlled experiment is terminated before an end of the duration of the controlled experiment.

12. The method of claim 1 , wherein the prediction model is one of a gradient boosting decision tree model and a linear regression model.

13. The method of claim 1 , wherein at least one feature derived from the behavior data is received and processed by the prediction model to execute the calculating of the predicted values of the performance parameter.

14. The method of claim 13 , wherein the at least one feature derived from the behavior data is one of: a total feature, a time series feature, a statistics feature, a periodicity feature and a derivative feature.

15. A hardware server comprising a hardware processor and computer-readable instructions for conducting a controlled experiment using prediction of future user behavior, the hardware processor being configured to:

collect behavior data on a first set of users and a second set of users over a first period of time, wherein:

the first set of users is exposed to a control variant of a service;

the second set of users is exposed to a treatment variant of the service; and

the behavior data relates to a performance parameter of the controlled experiment;

based on at least one prediction model applied to the behavior data, calculate predicted values of the performance parameter for each user of the first set of users and each user of the second set of users over the second period of time;

obtain actual values of the performance parameter for each user of the first set of users and each user of the second set of users based on the behavior data collected during the first period of time;

combine each actual value of the performance parameter for each user of the first set of users with the corresponding predicted value of the performance parameter for each user of the first set of users into a combined value of the performance parameter for each user of the first set of users;

combine each actual value of the performance parameter for each user of the second set of users with the corresponding predicted value of the performance parameter for each user of the second set of users into a combined value of the performance parameter for each user of the second set of users;

calculate an average value of the combined values of the performance parameter for each user of the first set of users;

calculate an average value of the combined values of the performance parameter for each user of the second set of users; and

determine if a statistically significant difference exists between the average value of the combined values of the performance parameter for each user of the first set of users and the average value of the combined values of the performance parameter for each user of the second set of users;

select, based on the statistically significant difference, one of the control variant of the service and the treatment variant of the service.

16. The server of claim 15 , wherein at least one feature derived from the behavior data is received and processed by the prediction model to execute the calculating of the predicted values of the performance parameter.