Network-aware product rollout in online social networks
In one embodiment, a method includes sending, from an application server to a first set of client systems, instructions for providing a treatment to a first set of users. The method includes determining an average treatment effect of the treatment for the first set of users based on information about user interactions associated with the treatment and a network exposure to the treatment for each user. Network exposure for a user is a measure of other users associated with the user that have been provided the treatment. The method includes modifying the treatment based on the determined average treatment effect. The method includes sending, to a second set of client systems, instructions for providing the modified treatment to a second set of users. The second set of users is a discrete, non-overlapping set of users with respect to the first set.
1. A method comprising, by one or more processors associated with one or more computing devices:
sending, from an application server to a first set of client systems, instructions for providing a treatment to a first set of users associated with the first set of client systems;
determining, by one or more processors of the application server, an average treatment effect of the treatment for the first set of users, wherein the average treatment effect is a function of (1) individual treatment effects for the first set of users based on, for each user in the first set of users, user interactions by the respective user with the treatment independent of whether other users in the first set of users are exposed to the treatment, and (2) a network treatment effect for the first set of users based on, for each user in the first set of users, a measure of social interference on the user interactions by the respective user with the treatment by other users in the first set of users exposed to the treatment;
modifying, by the one or more processors of the application server, the treatment based on the determined average treatment effect; and
sending, from the application server to a second set of client systems, instructions for providing the modified treatment to a second set of users associated with the second set of client systems, wherein the second set of users is a discrete, non-overlapping set of users with respect to the first set of users.
2. The method of claim 1 , wherein the average treatment effect is determined with an inverse probability weighting estimator.
3. The method of claim 1 , wherein the network treatment effect is further based on, for each user in the first set of users, a specified distribution ranging from a threshold level of network exposure to a maximum level of network exposure.
4. The method of claim 1 , wherein the network treatment effect is based on an exposure model comprising one or more of a full neighborhood exposure, an absolute k-neighborhood exposure, or a fractional q-neighborhood exposure.
5. The method of claim 1 , wherein determining the average treatment effect of the treatment is based in part on different levels of network exposure of a specified distribution range.
6. The method of claim 1 , wherein determining the average treatment effect, {circumflex over (τ)}, comprises determining
τ
^
(
Z
)
=
1
n
∑
i
=
1
n
(
Y
i
(
Z
)
1
[
Z
∈
σ
0
i
]
Pr
(
Z
∈
σ
0
i
)
-
Y
i
(
Z
)
1
[
Z
∈
σ
1
i
]
Pr
(
Z
∈
σ
1
i
)
)
,
and wherein:
Z represents a random vector comprising the values of {0, 1} n ;
n is a number of nodes in a particular cluster;
σ 1 i represents an exposure condition of {right arrow over (z)}={right arrow over (1)}, wherein {right arrow over (z)}={right arrow over (1)} represents a user in a treatment group;
σ 0 i represents the exposure condition of {right arrow over (z)}={right arrow over (0)}, wherein {right arrow over (z)}′={right arrow over (0)} represents a user in a control group;
i represents a user associated with a node;
Y i (Z) is a potential outcome of user i under a treatment assignment vector Z;
Pr(Z∈σ 1 i ) is a probability of network exposure to the treatment; and
Pr(Z∈σ 0 i ) is a probability of network exposure to the control.
7. The method of claim 1 , wherein the treatment and modified treatment modify the behavior of the first set of client systems and second set of client systems, respectively.
8. The method of claim 1 , wherein modifying the treatment comprises optimizing the user's experience or interaction with the treatment.
9. The method of claim 1 , further comprising:
modifying the modified treatment one or more additional times until the treatment is completed for a final launch, wherein the final launch comprises providing the treatment to all users of a product.
10. The method of claim 9 , wherein the product is an online social network.
11. The method of claim 1 , wherein the users are users of an online social network comprising a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, each of the edges between two of the nodes representing a single degree of separation between them, the plurality of nodes corresponding to a plurality of users associated with the online social network.
12. The method of claim 11 , wherein a node is network exposed if a threshold fraction of nodes within one degree of separation of the node are associated with the same treatment condition.
13. The method of claim 11 , further comprising:
identifying from the plurality of nodes a first cluster of nodes in the social graph; and
identifying from the plurality of nodes, a second cluster of nodes in the social graph using graph clustering, the second cluster comprising a discrete, non-overlapping set of nodes with respect to the first set of nodes.
14. The method of claim 13 , wherein the first cluster of nodes corresponds to the first set of users and the second cluster of nodes corresponds to the second set of users.
15. The method of claim 1 , wherein the treatment and modified treatment are a first and second version of a particular advertisement.
16. The method of claim 1 , wherein the treatment and modified treatment are a first and second version of a particular product or feature of an online social network or a third-party system.
17. The method of claim 1 , further comprising:
determining, by the one or more processors of the application server, a modified treatment effect of the modified treatment for the users of the second set of users; and
sending, from the application server to a third set of client systems, instructions for providing an updated treatment to a third set of users associated with the third set of client systems, wherein the updated treatment is based on the determined modified treatment effect.
18. The method of claim 1 , wherein a treatment effect of the treatment, {right arrow over (z)}, having n treatment conditions, {right arrow over (z)}={0, 1} n , is calculated for a user, i, based on a comparison of the user's behavior given exposure to a treatment condition Y i ({right arrow over (z)} n =1), the user's behavior given exposure to a control condition Y i ({right arrow over (z)} n =0), a probability of a network treatment effect in response to exposure to the treatment condition Pr i ({right arrow over (z)} n =1), and a probability of a network treatment effect in response to exposure to the control condition Pr i ({right arrow over (z)} n =0).
19. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
send, from an application server to a first set of client systems, instructions for providing a treatment to a first set of users associated with the first set of client systems;
determine, by one or more processors of the application server, an average treatment effect of the treatment for the first set of users, wherein the average treatment effect is a function of (1) individual treatment effects for the first set of users based on, for each user in the first set of users, user interactions by the respective user with the treatment independent of whether other users in the first set of users are exposed to the treatment, and (2) a network treatment effect for the first set of users based on, for each user in the first set of users, a measure of social interference on the user interactions by the respective user with the treatment by other users in the first set of users exposed to the treatment;
modify, by the one or more processors of the application server, the treatment based on the determined average treatment effect; and
send, from the application server to a second set of client systems, instructions for providing the modified treatment to a second set of users associated with the second set of client systems, wherein the second set of users is a discrete, non-overlapping set of users with respect to the first set of users.
20. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
send, from an application server to a first set of client systems, instructions for providing a treatment to a first set of users associated with the first set of client systems;
determine, by one or more processors of the application server, an average treatment effect of the treatment for the first set of users, wherein the average treatment effect is a function of (1) individual treatment effects for the first set of users based on, for each user in the first set of users, user interactions by the respective user with the treatment independent of whether other users in the first set of users are exposed to the treatment, and (2) a network treatment effect for the first set of users based on, for each user in the first set of users, a measure of social interference on the user interactions by the respective user with the treatment by other users in the first set of users exposed to the treatment;
modify, by the one or more processors of the application server, the treatment based on the determined average treatment effect; and
send, from the application server to a second set of client systems, instructions for providing the modified treatment to a second set of users associated with the second set of client systems, wherein the second set of users is a discrete, non-overlapping set of users with respect to the first set of users.