Identifying direct and indirect effects in A/B tests

1. A method performed by one or more data processing apparatus, the method comprising:

conducting an A/B test on a target element of an online platform, comprising:

selecting a measured mediator element of the online platform from a set of possible mediator elements of the online platform;

identifying a plurality of other mediator elements from the set of possible mediator elements as unmeasured mediator elements;

splitting a population of users of the online platform into at least a control group of users and a test group of users;

presenting a control version of the target element to the control group of users and presenting a test version of the target element to the test group of users; and

for each user in the population of users, measuring: (i) an outcome of the user interacting with the online platform, and (ii) an interaction of the user with the measured mediator element of the online platform, while refraining from measuring interaction of the user with the unmeasured mediator elements;

determining a direct effect and an indirect effect of the test version of the target element on user outcomes as mediated by the measured mediator element, comprising:

determining, based on the A/B test, a direct effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused independently of induced changes in user interaction with the measured mediator element; and

determining, based on the A/B test, an indirect effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused by induced changes in user interaction with the measured mediator element;

wherein the direct effect value and the indirect effect value do not depend on data measuring user interaction with the unmeasured mediator elements.

2. The method of claim 1 , wherein determining the direct effect value and the indirect effect value comprises:

identifying values of coefficients of a system of equations based on the A/B test, wherein the system of equations comprises:

a first equation that expresses user interaction with the measured mediator element as a function of whether the control version or the test version of the target element was presented, and

a second equation that expresses the outcome of user interaction with the online platform as a function of: (i) whether the control version or the test version of the target element was presented, and (ii) user interaction with the measured mediator element; and

determining the direct effect value and the indirect effect value based on the values of the coefficients of the system of equations.

3. The method of claim 2 , wherein the first equation and the second equation are linear equations.

4. The method of claim 3 , wherein:

the first equation is given by:

M=θ M,0 +θ M,1 ·T+μ M

wherein M specifies user interaction with the measured mediator element, T specifies whether the control version or the test version of the target element was presented, and θ M,0 , θ M,1 , and μ M are coefficients of the system of equations; and

the second equation is given by:

Y=θ Y,0 +θ Y,1 ·T+θ Y,2 ·M+θ Y,3 ·M·T+μ Y

wherein Y specifies the outcome of user interaction with the online platform, T specifies whether the control version or the test version of the target element was presented, M specifies user interaction with the measured mediator element, and θ Y,0 , θ Y,1 , θ Y,2 , θ Y,3 , and μ Y are coefficients of the system of equations.

5. The method of claim 2 , wherein identifying the values of the coefficients of the system of equations comprises using an ordinary least squares technique.

6. The method of claim 2 , wherein identifying the values of the coefficients of the system of equations comprises using an iterative general method of moments (ITGMM) technique.

7. The method of claim 4 , wherein determining the direct effect value and the indirect effect value based on the values of the coefficients of the system of equations comprises:

determining the direct effect value DE as:

DE=θ Y,1 +θ Y,3 (θ M,0 +θ M,1 ·t ),

wherein t=0 or t=1, and

determining the indirect effect value IE as:

IE=θ M,1 (θ Y,2 +θ Y,2 ·t ),

wherein t=0 or t=1.

8. The method of claim 1 , further comprising determining a statistical significance of the direct effect value and determining a statistical significance of the indirect effect value.

9. The method of claim 1 , wherein the online platform comprises an e-commerce web site.

10. The method of claim 1 , wherein splitting the population of users of the online platform into at least the control group of users and the test group of users comprises randomly splitting the population of users of the online platform into at least the control group of users and the test group of users.

11. The method of claim 1 , wherein the target element of the online platform comprises a recommendation module.

12. The method of claim 1 , wherein the measured mediator element of the online platform comprises a search module.

13. The method of claim 1 , wherein the measured mediator element of the online platform comprises a promoted listings module.

14. The method of claim 1 , wherein the outcome of the user interacting with the online platform comprises a conversion status of the user or a value of products purchased by the user on the online platform.

15. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

conducting an A/B test on a target element of an online platform, comprising:

selecting a measured mediator element of the online platform from a set of possible mediator elements of the online platform;

identifying a plurality of other mediator elements from the set of possible mediator elements as unmeasured mediator elements;

splitting a population of users of the online platform into at least a control group of users and a test group of users;

presenting a control version of the target element to the control group of users and presenting a test version of the target element to the test group of users; and

for each user in the population of users, measuring: (i) an outcome of the user interacting with the online platform, and (ii) an interaction of the user with the measured mediator element of the online platform, while refraining from measuring interaction of the user with the unmeasured mediator elements;

determining a direct effect and an indirect effect of the test version of the target element on user outcomes as mediated by the measured mediator element, comprising:

determining, based on the A/B test, a direct effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused independently of induced changes in user interaction with the measured mediator element; and

determining, based on the A/B test, an indirect effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused by induced changes in user interaction with the measured mediator element;

wherein the direct effect value and the indirect effect value do not depend on data measuring user interaction with the unmeasured mediator elements.

16. The system of claim 15 , wherein determining the direct effect value and the indirect effect value comprises:

identifying values of coefficients of a system of equations based on the A/B test, wherein the system of equations comprises:

a first equation that expresses user interaction with the measured mediator element as a function of whether the control version or the test version of the target element was presented, and

a second equation that expresses the outcome of user interaction with the online platform as a function of: (i) whether the control version or the test version of the target element was presented, and (ii) user interaction with the measured mediator element; and

determining the direct effect value and the indirect effect value based on the values of the coefficients of the system of equations.

17. The system of claim 16 , wherein the first equation and the second equation are linear equations.

18. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

conducting an A/B test on a target element of an online platform, comprising:

selecting a measured mediator element of the online platform from a set of possible mediator elements of the online platform;

identifying a plurality of other mediator elements from the set of possible mediator elements as unmeasured mediator elements;

splitting a population of users of the online platform into at least a control group of users and a test group of users;

presenting a control version of the target element to the control group of users and presenting a test version of the target element to the test group of users; and

for each user in the population of users, measuring: (i) an outcome of the user interacting with the online platform, and (ii) an interaction of the user with the measured mediator element of the online platform, while refraining from measuring interaction of the user with the unmeasured mediator elements;

determining a direct effect and an indirect effect of the test version of the target element on user outcomes as mediated by the measured mediator element, comprising:

determining, based on the A/B test, a direct effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused independently of induced changes in user interaction with the measured mediator element; and

determining, based on the A/B test, an indirect effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused by induced changes in user interaction with the measured mediator element;

wherein the direct effect value and the indirect effect value do not depend on data measuring user interaction with the unmeasured mediator elements.

19. The non-transitory computer storage media of claim 18 , wherein determining the direct effect value and the indirect effect value comprises:

identifying values of coefficients of a system of equations based on the A/B test, wherein the system of equations comprises:

a first equation that expresses user interaction with the measured mediator element as a function of whether the control version or the test version of the target element was presented, and

a second equation that expresses the outcome of user interaction with the online platform as a function of: (i) whether the control version or the test version of the target element was presented, and (ii) user interaction with the measured mediator element; and

determining the direct effect value and the indirect effect value based on the values of the coefficients of the system of equations.

20. The non-transitory computer storage media of claim 19 , wherein the first equation and the second equation are linear equations.