Method and system for automatic detection and prevention of quality issues in online experiments
The present teaching relates to managing online experiments. In one example, a plurality of experiment layers is created with respect to a plurality of online users. Each experiment layer includes at least one experiment each of which includes one or more buckets associated with respective features to be experimented on. Each of the plurality of online users is assigned to a corresponding bucket in each experiment layer, such that the user is simultaneously associated with multiple experiments in different layers. User event data related to the plurality of experiment layers are collected from the plurality of online users. One or more contaminated buckets are automatically detected based on the user event data.
1. A method, implemented on at least one computing device each of which has at least one processor, storage, and a communication platform connected to a network for managing online experiments, the method comprising:
creating, by an online controlled experimentation engine, a plurality of experiment layers, wherein each experiment layer includes and runs at least one online experiment on a website, and each online experiment includes one or more buckets associated with one or more features to be experimented on;
assigning, by the online controlled experimentation engine, via the website embedded with a randomization algorithm on a load balancer, by splitting traffic between the one or more buckets, based on a bucket size, a plurality of identifiers, each of which corresponds to one of a plurality of online users, to a corresponding bucket in each experiment layer such that a user is simultaneously associated with multiple online experiments in different experiment layers;
collecting user event data related to the plurality of experiment layers from the plurality of online users;
performing a two-stage test to determine health of the plurality of experiment layers, wherein the two-stage test comprises:
automatically detecting one or more contaminated experiment layers from the plurality of experiment layers by computing, based on the user event data, a uniform distribution test for each of the plurality of experiment layers, and
in response to the one or more contaminated experiment layers from the plurality of experiment layers being detected, automatically detecting, for each of the detected one or more contaminated experiment layers, one or more contaminated buckets by computing, based on the user event data, a proportion test of identifiers assigned to each bucket in each of the one or more contaminated experiment layers; and
in response to determining that a severity level of at least one of the one or more contaminated buckets exceeds a threshold severity level, performing, by the online controlled experimentation engine, contamination correction by:
automatically removing the at least one of the one or more contaminated buckets from a corresponding online experiment to prevent the at least one of the one or more contaminated buckets from contaminating the corresponding online experiment, and
reinstating the bucket size in the corresponding online experiment.
2. The method of claim 1 , wherein the plurality of identifiers comprise a plurality of experiment unit identifiers (IDs), the step of assigning the plurality of identifiers comprises:
determining, for each of the plurality of online users, a corresponding experiment unit ID of the plurality of experiment unit IDs;
determining a random seed for each experiment layer;
calculating, for each experiment layer, a hash value associated with each of the plurality of identifiers based on the corresponding experiment unit ID and the random seed; and
assigning the identifier to the corresponding bucket in each experiment layer based on the calculated hash value.
3. The method of claim 1 , wherein the step of automatically detecting the one or more contaminated experiment layers comprises:
calculating a health level of each experiment layer based on the uniform distribution test which is computed for each experiment layer based on the user event data;
determining one or more unhealthy experiment layers based on the health level of each experiment layer; and
determining, for each of the one or more unhealthy experiment layers, at least one contaminated bucket, wherein the one or more contaminated buckets comprise the at least one contaminated bucket determined in each of the one or more unhealthy experiment layers.
4. The method of claim 3 , wherein the uniform distribution test is used to determine whether a traffic distribution across different buckets in a given experiment layer is uniform.
5. The method of claim 3 , wherein the proportion test is used to determine a probability of assigning an online user to each bucket in each experiment layer, and the proportion test identifies problematic hash ranges for each of the detected one or more contaminated experiment layers.
6. The method of claim 1 , further comprising:
selecting a prevention scheme based on each of the one or more contaminated buckets that were detected;
determining the severity level for each of the one or more contaminated buckets; and
determining a prevention action to be applied to the at least one of the one or more contaminated buckets based on the prevention scheme and the at least one of the one or more contaminated buckets exceeding the threshold severity level, wherein the prevention action comprises:
automatically removing the at least one of the one or more contaminated buckets from the corresponding online experiment, and
after the at least one of the one or more contaminated buckets is removed, reinstating the bucket size in the corresponding online experiment.
7. The method of claim 6 , wherein each experiment layer comprises a plurality of online experiments, and wherein the prevention action further comprises:
holding or closing the corresponding online experiment when a condition related to all contaminated buckets in the corresponding online experiment is met.
8. A system having at least one processor, storage, and a communication platform connected to a network for managing online experiments, comprising:
a multi-layer experiments creator of an online controlled experimentation engine configured for creating a plurality of experiment layers, wherein each experiment layer includes and runs at least one online experiment on a website, and each online experiment includes one or more buckets associated with one or more features to be experimented on;
a user assigner of the online controlled experimentation engine configured for assigning, via the website embedded with a randomization algorithm on a load balancer, by splitting traffic between the one or more buckets, based on a bucket size, a plurality of identifiers, each of which corresponds to one of a plurality of online users, to a corresponding bucket in each experiment layer such that a user is simultaneously associated with multiple online experiments in different experiment layers;
a grid based event analyzer configured for collecting user event data related to the plurality of experiment layers from the plurality of online users; and
an experiment contamination detector configured for:
performing a two-stage test to determine health of the plurality of experiment layers, wherein the two-stage test comprises:
automatically detecting one or more contaminated experiment layers from the plurality of experiment layers by computing, based on the user event data, a uniform distribution test for each of the plurality of experiment layers, and
in response to the one or more contaminated experiment layers from the plurality of experiment layers being detected, automatically detecting, for each of the detected one or more contaminated experiment layers, one or more contaminated buckets by computing, based on the user event data, a proportion test of identifiers assigned to each bucket in each of the one or more contaminated experiment layers; and
a prevention action determiner of the online controlled experimentation engine configured for performing contamination correction by:
in response to determining that a severity level of at least one of the one or more contaminated buckets exceeds a threshold severity level, automatically removing the at least one of the one or more contaminated buckets from a corresponding online experiment to prevent the at least one of the one or more contaminated buckets from contaminating the corresponding online experiment, and
reinstating the bucket size in the corresponding online experiment.
9. The system of claim 8 , wherein the plurality of identifiers comprise a plurality of experiment unit identifiers (IDs), the user assigner comprises:
an experiment unit ID determiner configured for determining, for each of the plurality of online users, a corresponding experiment unit ID of the plurality of experiment unit IDs;
a random seed determiner configured for determining a random seed for each experiment layer;
a hash value calculator configured for calculating, for each experiment layer, a hash value associated with each of the plurality of identifiers based on the corresponding experiment unit ID and the random seed; and
a bucket assigner configured for assigning the identifier to the corresponding bucket in each experiment layer based on the calculated hash value.
10. The system of claim 8 , wherein the experiment contamination detector comprises:
a test statistics calculator configured for calculating a health level of each experiment layer based on the uniform distribution test which is computed for each experiment layer based on the user event data;
a uniformity tester configured for determining one or more unhealthy experiment layers based on the health level of each experiment layer; and
a proportion tester configured for performing the proportion test to determine, for each of the one or more unhealthy experiment layers, at least one contaminated bucket, wherein the one or more contaminated buckets comprise the at least one contaminated bucket determined in each of the one or more unhealthy experiment layers.
11. The system of claim 10 , wherein the uniform distribution test is used to determine whether a traffic distribution across different buckets in a given experiment layer is uniform.
12. The system of claim 10 , wherein the proportion test is used to determine a probability of assigning an online user to each bucket in each experiment layer, and the proportion test identifies problematic hash ranges for each of the detected one or more contaminated experiment layers.
13. The system of claim 8 , further comprising:
a prevention scheme selector configured for selecting a prevention scheme based on each of the one or more contaminated buckets that were detected; and
a severity level determiner configured for determining the severity level for each of the one or more contaminated buckets;
wherein the prevention action determiner is configured for determining a prevention action to be applied to the at least one of the one or more contaminated buckets based on the prevention scheme and the at least one of the one or more contaminated buckets exceeding the threshold severity level, wherein the prevention action comprises:
automatically removing the at least one of the one or more contaminated buckets from the corresponding online experiment, and
after the at least one of the one or more contaminated buckets is removed, reinstating the bucket size in the corresponding online experiment.
14. The system of claim 13 , wherein each experiment layer comprises a plurality of online experiments, and wherein the prevention action further comprises:
holding or closing the corresponding online experiment when a condition related to all contaminated buckets in the corresponding online experiment is met.
15. A non-transitory machine-readable medium having information recorded thereon for managing online experiments, wherein the information, when read by an online controlled experimentation engine, causes the online controlled experimentation engine to perform operations comprising:
creating a plurality of experiment layers, wherein each experiment layer includes and runs at least one online experiment on a website, and each online experiment includes one or more buckets associated with one or more features to be experimented on;
assigning, via the website embedded with a randomization algorithm on a load balancer, by splitting traffic between the one or more buckets, based on a bucket size, a plurality of identifiers, each of which corresponds to one of a plurality of online users, to a corresponding bucket in each experiment layer such that a user is simultaneously associated with multiple online experiments in different experiment layers;
collecting user event data related to the plurality of experiment layers from the plurality of online users;
performing a two-stage test to determine health of the plurality of experiment layers, wherein the two-stage test comprises:
automatically detecting one or more contaminated experiment layers from the plurality of experiment layers by computing, based on the user event data, a uniform distribution test for each of the plurality of experiment layers, and
in response to the one or more contaminated experiment layers from the plurality of experiment layers being detected, automatically detecting, for each of the detected one or more contaminated experiment layers, one or more contaminated buckets by computing, based on the user event data, a proportion test of identifiers assigned to each bucket in each of the one or more contaminated experiment layers; and
in response to determining that a severity level of at least one of the one or more contaminated buckets exceeds a threshold severity level, performing contamination correction by:
automatically removing the at least one of the one or more contaminated buckets from a corresponding online experiment to prevent the at least one of the one or more contaminated buckets from contaminating the corresponding online experiment, and
reinstating the bucket size in the corresponding online experiment.
16. The medium of claim 15 , wherein the plurality of identifiers comprise a plurality of experiment unit identifiers (IDs), the step of assigning the plurality of identifiers comprises:
determining, for each of the plurality of online users, a corresponding experiment unit ID of the plurality of experiment unit IDs;
determining a random seed for each experiment layer;
calculating, for each experiment layer, a hash value associated with each of the plurality of identifiers based on the corresponding experiment unit ID and the random seed; and
assigning the identifier to the corresponding bucket in each experiment layer based on the calculated hash value.
17. The medium of claim 15 , wherein the step of automatically detecting comprises:
calculating a health level of each experiment layer based on the uniform distribution test which is computed for each experiment layer based on the user event data;
determining one or more unhealthy experiment layers based on the health level of each experiment layer; and
determining, for each of the one or more unhealthy experiment layers, at least one contaminated bucket, wherein the one or more contaminated buckets comprise the at least one contaminated bucket determined in each of the one or more unhealthy experiment layers.
18. The medium of claim 17 , wherein the uniform distribution test is used to determine whether a traffic distribution across different buckets in a given experiment layer is uniform.
19. The medium of claim 17 , wherein the proportion test is used to determine a probability of assigning an online user to each bucket in each experiment layer, and the proportion test identifies problematic hash ranges for each of the detected one or more contaminated experiment layers.
20. The medium of claim 15 , wherein the operations further comprise:
selecting a prevention scheme based on each of the one or more contaminated buckets that were detected;
determining the severity level for each of the one or more contaminated buckets; and
determining a prevention action to be applied to the at least one of the one or more contaminated buckets based on the prevention scheme and the at least one of the one or more contaminated buckets exceeding the threshold severity level, wherein the prevention action comprises:
automatically removing the at least one of the one or more contaminated buckets from the corresponding online experiment, and
after the at least one of the one or more contaminated buckets is removed, reinstating the bucket size in the corresponding online experiment.
21. The medium of claim 20 , wherein each experiment layer comprises a plurality of online experiments, and wherein the prevention action further comprises:
holding or closing the corresponding online experiment when a condition related to all contaminated buckets in the corresponding online experiment is met.