BIAS CORRECTION AND ESTIMATION IN NETWORK A/B TESTING

1 . A method, comprising:

obtaining, for a set of users in a social network, a set of treatment assignments of the users in an A/B test, wherein the treatment assignments indicate exposure of the users to a control version or a treatment version of a message;

obtaining, for each of the users, a fraction of neighbors exposed to the treatment version in the A/B test;

applying, by a computer system, a statistical model to the treatment assignments and the fraction of neighbors exposed to the treatment version to estimate an average treatment effect (ATE) for the set of users;

selecting, based on the ATE, a fraction of additional users in the social network for subsequent exposure to the treatment version; and

presenting the treatment version to the fraction of additional users.

2 . The method of claim 1 , further comprising:

applying the statistical model to a set of responses of the users to the treatment version and the control version to estimate the ATE.

3 . The method of claim 2 , wherein applying the statistical model to the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses of the users to estimate the ATE comprises:

using the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses to estimate a global bias, a treatment effect, and a network effect in the statistical model; and

using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE.

4 . The method of claim 3 , wherein an ordinary least squares technique is used to estimate the global bias, the treatment effect, or the network effect.

5 . The method of claim 3 , wherein using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE comprises:

using the estimated treatment effect and the estimated network effect to estimate the ATE.

6 . The method of claim 3 , wherein using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE comprises:

using the estimated global bias for users exposed to the treatment version, the estimated global bias for users exposed to the control version, and the estimated network effect for users exposed to the treatment version to estimate the ATE.

7 . The method of claim 1 , wherein obtaining the set of treatment assignments of the users in the A/B test comprises:

calculating a set of equally sized clusters of the users in the social network; and

randomly selecting a subset of the equally sized clusters for exposure to the treatment version during the A/B test.

8 . The method of claim 7 , wherein calculating the set of equally sized clusters of the users in the social network comprises:

iteratively swapping memberships of the users among the equally sized clusters to increase a number of edges in each of the equally sized clusters.

9 . The method of claim 1 , wherein the fraction of neighbors exposed to the treatment version in the A/B test is obtained using a graph of the social network.

10 . The method of claim 1 , wherein the statistical model comprises a regression model.

11 . An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

obtain, for a set of users in a social network, a set of treatment assignments of the users in an A/B test, wherein the treatment assignments indicate exposure of the users to a control version or a treatment version of a message;

obtain, for each of the users, a fraction of neighbors exposed to the treatment version in the A/B test;

apply a statistical model to the treatment assignments and the fraction of neighbors exposed to the treatment version to estimate an average treatment effect (ATE) for the set of users;

select, based on the ATE, a fraction of additional users in the social network for subsequent exposure to the treatment version; and

present the treatment version to the fraction of additional users.

12 . The apparatus of claim 11 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

applying the statistical model to a set of responses of the users to the treatment version and the control version to estimate the ATE.

13 . The apparatus of claim 12 , wherein applying the statistical model to the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses to estimate the ATE comprises:

using the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses to estimate a global bias, a treatment effect, and a network effect in the statistical model; and

using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE.

14 . The apparatus of claim 12 , wherein using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE comprises:

using the estimated treatment effect and the estimated network effect to estimate the ATE.

15 . The apparatus of claim 12 , wherein using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE comprises:

using the estimated global bias for users exposed to the treatment version, the estimated global bias for users exposed to the control version, and the estimated network effect for users exposed to the treatment version to estimate the ATE.

16 . The apparatus of claim 11 , wherein the statistical model comprises a regression model.

17 . A system comprising:

a sampling non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the system to obtain, for a set of users in a social network, a set of treatment assignments of the users in an A/B test, wherein the treatment assignments indicate exposure of the users to a control version or a treatment version of a message; and

an estimation non-transitory computer readable medium comprising instructions that, when executed by the one or more processors, cause the system to:

obtain, for each of the users, a fraction of neighbors exposed to the treatment version in the A/B test;

obtain a set of responses of the users to the treatment version and the control version;

apply a statistical model to the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses of the users to estimate an average treatment effect (ATE) for the set of users;

select, based on the ATE, a fraction of additional users in the social network for subsequent exposure to the treatment version; and

present the treatment version to the fraction of additional users.

18 . The system of claim 17 , wherein applying the statistical model to the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses of the users to estimate the ATE comprises:

using the treatment assignments, the fraction of neighbors exposed to the treatment version, and the responses to estimate a global bias, a treatment effect, and a network effect in the statistical model; and

using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE.

19 . The system of claim 18 , wherein using the estimated global bias, the estimated treatment effect, or the estimated network effect in the statistical model to estimate the ATE comprises at least one of:

using the estimated treatment effect and the estimated network effect to estimate the ATE; and

using the estimated global bias for users exposed to the treatment version, the estimated global bias for users exposed to the control version, and the estimated network effect for users exposed to the treatment version to estimate the ATE.

20 . The system of claim 17 , wherein obtaining the set of treatment assignments of the users in the A/B test comprises:

calculating a set of equally sized clusters of the users in the social network; and

randomly selecting a subset of the equally sized clusters for exposure to the treatment version during the A/B test.