POST EXPERIMENT POWER
Techniques for conducting A/B experimentation of online content are described. According to various embodiments, a user specification of a metric being recorded as a result of an online A/B experiment of online content is received, the online A/B experiment being targeted at a segment of members of an online social networking service. Thereafter, a power value for the A/B experiment that is associated with the metric is calculated, the power value indicating an inferred ability to detect changes in a value of the metric during performance of the A/B experiment. The power value for the A/B experiment is then displayed via a user interface displayed on a client device.
1 . A method comprising:
receiving, by at least one hardware processor, a user specification of a metric being recorded as a result of an online A/B experiment of online content, the online A/B experiment being targeted at a segment of members of an online social networking service;
calculating, by at least one hardware processor, a power value for the A/B experiment that is associated with the metric, the power value indicating an inferred ability to detect changes in a value of the metric during performance of the A/B experiment; and
transmitting, by the at least one hardware processor, the power value for the A/B experiment to be displayed on a user interface displayed on a client device.
2 . The method of claim 1 , wherein the calculating further comprises:
generating, based on results of prior A/B experiments, a computer-based model associated with the metric, the model indicating trends in the value of the metric over time during the prior A/B experiments;
applying present values of the metric for each variant of the A/B experiment to the model to determine future values of the metric for each variant of the A/B experiment; and
determining the power value, based on the determined future values of the metric for each variant of the A/B experiment.
3 . The method of claim 1 , further comprising:
comparing the calculated power value to a specific power value threshold;
determining, based on the comparison, that the power value for the A/B experiment is not sufficient for detecting changes in the value of the metric during performance of the A/B experiment; and
displaying, via the user interface displayed on the client device, a notification that the power value for the A/B experiment is not sufficient for detecting changes in the value of the metric during performance of the A/B experiment.
4 . The method of claim 1 , further comprising:
identifying a modification to the online A/B experiment to improve the power value; and
displaying, via the user interface displayed on the client device, a recommendation of the modification to the online A/B experiment.
5 . The method of claim 4 , wherein the recommendation is to extend a duration of the online A/B experiment for a specific time interval.
6 . The method of claim 5 , wherein the identifying further comprises:
generating, based on results of prior A/B experiments, a computer-based model associated with the metric, the model indicating trends in the value of the metric over time during the prior A/B experiments;
applying present values of the metric for each variant of the A/B experiment to the model to determine future values of the metric for each variant of the A/B experiment;
calculating, for each specific date in a range of future dates, based on the future values for the specific date, a future power value for the A/B experiment that is associated with the metric, the future power value indicating the inferred ability to detect changes in a value of the metric during performance of the A/B experiment on the specific date;
identifying a particular date in the range of future dates associated with a highest future power value; and
determining that the specific time interval has an end date corresponding to the particular date.
7 . The method of claim 4 , wherein the recommendation is to initiate a new A/B experiment wherein a particular variant of the online A/B experiment that is ramped to a particular percentage of the targeted segment of members during the online A/B experiment is ramped to a new percentage of the targeted segment of members in the new A/B experiment.
8 . The method of claim 1 , wherein the metric corresponds to a number of page views, a number of unique users, a number of clicks, or a click through rate.
9 . The method of claim 1 , wherein the power value corresponds to a percentage value.
10 . The method of claim 1 , further comprising receiving a user specification of a minimal detectable event value,
wherein the power value for the A/B experiment indicates an inferred ability to detect changes in the value of the metric greater than the minimal detectable event value during performance of the A/B experiment.
11 . The method of claim 10 , wherein the calculating further comprises:
generating, based on results of prior A/B experiments, a computer-based model associated with the metric, the model indicating trends in the value of the metric over time during the prior A/B experiments;
applying present values of the metric for each variant of the A/B experiment to the model to determine future values of the metric for each variant of the A/B experiment;
determining that a degree of change greater than the minimal detectable event value exists between the future values and the present values for each variant of the A/B experiment; and
determining the power value, based on the degree of change for each variant of the A/B experiment.
12 . The method of claim 1 , wherein the received user specification specifies a plurality of metrics including the metric, and wherein the calculated power value is associated with the plurality of metrics including the metric, the power value indicating an inferred ability to detect changes in a value of one or more of the plurality of metrics during performance of the A/B experiment.
13 . The method of claim 12 , wherein the power value associated with the plurality of metrics is generated by:
calculating a plurality of metric-specific power values associated with the plurality of metrics; and
calculating the power value based on the plurality of metric-specific power values.
14 . A system comprising:
a processor; and
a memory device holding an instruction set executable on the processor to cause the system to perform operations comprising:
receiving a user specification of a metric being recorded as a result of an online A/B experiment of online content, the online A/B experiment being targeted at a segment of members of an online social networking service;
calculating a power value for the A/B experiment that is associated with the metric, the power value indicating an inferred ability to detect changes in a value of the metric during performance of the A/B experiment; and
displaying, via a user interface displayed on a client device, the power value for the A/B experiment.
15 . The system of claim 14 , wherein the calculating further comprises:
generating, based on results of prior A/B experiments, a computer-based model associated with the metric, the model indicating trends in the value of the metric over time during the prior A/B experiments;
applying present values of the metric for each variant of the A/B experiment to the model to determine future values of the metric for each variant of the A/B experiment; and
determining the power value, based on the determined future values of the metric for each variant of the A/B experiment.
16 . The system of claim 14 , further comprising:
comparing the calculated power value to a specific power value threshold;
determining, based on the comparison, that the power value for the A/B experiment is not sufficient for detecting changes in the value of the metric during performance of the A/B experiment; and
displaying, via the user interface displayed on the client device, a notification that the power value for the A/B experiment is not sufficient for detecting changes in the value of the metric during performance of the A/B experiment.
17 . The system of claim 14 , further comprising:
identifying a modification to the online A/B experiment to improve the power value; and
displaying, via the user interface displayed on the client device, a recommendation of the modification to the online A/B experiment.
18 . The system of claim 17 , wherein the recommendation is to extend a duration of the online A/B experiment for a specific time interval.
19 . The system of claim 17 , wherein the recommendation is to initiate a new A/B experiment wherein a particular variant of the online A/B experiment that is ramped to a particular percentage of the targeted segment of members during the online A/B experiment is ramped to a new percentage of the targeted segment of members in the new A/B experiment.
20 . A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving a user specification of a metric being recorded as a result of an online A/B experiment of online content, the online A/B experiment being targeted at a segment of members of an online social networking service;
calculating a power value for the A/B experiment that is associated with the metric, the power value indicating an inferred ability to detect changes in a value of the metric during performance of the A/B experiment; and
displaying, via a user interface displayed on a client device, the power value for the A/B experiment.