Evaluating The Efficacy Of An Advertisement Campaign
Methods for A/B testing to evaluate the efficacy of an advertisement campaign are disclosed. An advertisement request is received via an IP network from a publisher connected to the IP network. The advertisement request comprises information including a device identifier of a device requesting a resource from the publisher over the IP network. A determination is made to respond to the advertisement request or not on the basis of the information in the advertisement request. In response to determining to respond to the advertisement request, a hash of the device identifier is evaluated to determine whether the device belongs to a control group or not. In response to determining that the device belongs to the control group, no response is made to the advertisement request.
1 . A method comprising performing A/B testing to evaluate the efficacy of an advertisement campaign, in which advertisements are shown to users of devices connected to an Internet Protocol (IP) network, the method comprising:
receiving an advertisement request via the IP network from a publisher connected to the IP network, wherein the advertisement request comprises information including a device identifier of a device requesting a resource from the publisher over the IP network;
determining whether to respond to the advertisement request or not on the basis of the information in the advertisement request, and
in response to determining to respond to the advertisement request, evaluating a hash of the device identifier to determine whether the device belongs to a control group or not; and
in response to determining that the device belongs to the control group, not responding to the advertisement request.
2 . The method of claim 1 , in which the hash is evaluated by:
converting the device identifier into an integer;
adding a constant to the integer to produce a shifted integer; and
performing a modulo operation on the shifted integer to produce a remainder.
3 . The method of claim 2 , in which the modulo operation comprises evaluating the remainder of the shifted integer after division by 100.
4 . The method of claim 2 , in which the constant is a random integer that is specific to a particular advertisement campaign.
5 . The method of claim 2 , in which the determination as to whether the device belongs to the control group or not is made by:
comparing the remainder to a threshold;
in response to the comparison identifying that the remainder is below the threshold, identifying the device as belonging to the control group; and
in response to the comparison identifying that the remainder is not below the threshold, identifying the device as not belonging to the control group.
6 . The method of claim 1 , further comprising a step of logging occasions on which it is determined that the device belongs to the control group.
7 . The method of claim 1 , further comprising a step of tracking the location of the device so as to determine whether the advertisement influences users' location behavior.
8 . A non-transitory computer-readable medium having computer- readable instructions encoded thereon, in which said computer-readable instructions, when executed by a computer, cause the computer to perform a method comprising performing A/B testing to evaluate the efficacy of an advertisement campaign, in which advertisements are shown to users of devices connected to an Internet Protocol (IP) network, the method comprising:
receiving an advertisement request via the IP network from a publisher connected to the IP network, wherein the advertisement request comprises information including a device identifier of a device requesting a resource from the publisher over the IP network;
determining whether to respond to the advertisement request or not on the basis of the information in the advertisement request, and
in response to determining to respond to the advertisement request, evaluating a hash of the device identifier to determine whether the device belongs to a control group or not; and
in response to determining that the device belongs to the control group, not responding to the advertisement request.
9 . The non-transitory computer-readable medium of claim 8 , in which the hash is evaluated by:
converting the device identifier into an integer;
adding a constant to the integer to produce a shifted integer; and
performing a modulo operation on the shifted integer to produce a remainder.
10 . The non-transitory computer-readable medium of claim 9 , in which the modulo operation comprises evaluating the remainder of the shifted integer after division by 100.
11 . The non-transitory computer-readable medium of claim 9 , in which the constant is a random integer that is specific to a particular advertisement campaign.
12 . The non-transitory computer-readable medium of claim 9 , in which the determination as to whether the device belongs to the control group or not is made by:
comparing the remainder to a threshold;
in response to the comparison identifying that the remainder is below the threshold, identifying the device as belonging to the control group; and
in response to the comparison identifying that the remainder is not below the threshold, identifying the device as not belonging to the control group.
13 . The non-transitory computer-readable medium of claim 8 , in which said computer-readable instructions cause the computer to execute a further step of logging occasions on which it is determined that the device belongs to the control group.
14 . The non-transitory computer-readable medium of claim 8 , in which said computer-readable instructions cause the computer to execute a further step of tracking the location of the device so as to determine whether the advertisement influences users' location behavior