Determining variations of content to provide to users in variation testing of content
A variation testing system environment for performing variation testing of web pages and applications is disclosed. Users requesting a view from a content provider are not randomly assigned to one of a plurality of variations of the view. Rather, a function is applied to each user's identifier in order to determine which variation of the view is provided to a client device of the user.
1. A computer-implemented method of determining a variation of a web page to provide to client devices of a user, the method comprising:
receiving, from an application of a first client device of a user, an indication of a request by the first client device for a web page, the web page associated with a plurality of different variations of the web page;
determining an identifier associated with a user of the first client device; applying a function to the identifier to generate an assignment identifier for the user; and
wherein applying the function comprises:
generating a hash of the identifier using the function; and
normalizing the hash of the identifier to generate the assignment identifier that is within an interval, wherein the assignment identifier and other assignment identifiers associated with other users are distributed within the interval based on a proportion of corresponding users assigned to each of the plurality of different variations;
selecting a variation of the web page from the plurality of different variations of the web page that corresponds to the assignment identifier for the user; and
receiving, from an application of a second client device of the user that is a different client device type from the first client device of the user, an indication of a request by the second client device for the same web page requested by the first client device;
wherein the same selected variation of the web page is provided to both the first client device of the user and the second client device of the user.
2. The computer-implemented method of claim 1 , wherein the function applied to the identifier is a deterministic, uniform and pseudorandom number generator.
3. The computer-implemented method of claim 2 , further comprising: determining an identifier associated with a variation test for the web page that includes the plurality of different variations of the web page.
4. The computer-implemented method of claim 3 , wherein applying the function comprises:
applying the deterministic, uniform and pseudorandom generator to a concatenation of the identifier and the identifier associated with the variation test for the web page to generate the hash of the identifier.
5. The computer-implemented method of claim 4 , wherein selecting the variation of the web page comprises:
identifying a policy for the web page that describes which variation of the web page to provide based on assignment identifiers;
comparing the assignment identifier to the policy; and
selecting the variation of the web page to provide in response to receiving the identifier based on the comparison.
6. The computer-implemented method of claim 5 , wherein the plurality of different variations of the web page comprise only a first variation of the web page and a second variation of the web page, and the policy includes a threshold indicative of whether to provide the first variation or the second variation in response to receiving the identifier, and wherein comparing the assignment identifier to the policy comprises:
comparing the assignment identifier to the threshold;
responsive to the assignment identifier being below the threshold, selecting the first variation of the web page to provide in response to receiving the identifier; and
responsive to the assignment identifier being above the threshold, selecting the second variation of the web page to provide in response to receiving the identifier.
7. The computer-implemented method of claim 5 , wherein comparing the assignment identifier to the policy comprises:
identifying, from the policy, a mapping of a plurality of ranges of assignment identifiers to the plurality of variations of the web page, each range of assignment identifiers associated with a different variation of the web page;
identifying a range of assignment identifiers from the plurality of ranges of assignment identifiers that includes the assignment identifier; and
identifying a variation of the web page that is associated with the identified range of assignment identifiers.
8. The computer-implemented method of claim 1 , wherein receiving the indication of the request by the first client device for the web page comprises:
receiving, from a content provider, the identifier, wherein the content provider receives the request for the web page from the first client device.
9. The computer-implemented method of claim 1 , wherein providing the selected variation of the web page comprises:
transmitting one or more notifications to the content provider, the one or more notifications each including an indication of the selected variation of the web page to provide to the first client device and the second client device;
wherein the content provider provides the selected variation of the web page to the first client device and the second client device based on the notification.
10. The computer-implemented method of claim 1 , wherein receiving the indication comprises:
receiving, by a computer system, the request by the first client device for the web page.
11. The computer-implemented method of claim 1 , wherein the application is a web browser of the first client device and the first client device selects the variation of the web page.
12. A computer program product comprising a non-transitory computer-readable storage medium storing executable code of determining a variation of a web page to provide to client devices of a user, the code when executed causing a computer to perform operations comprising:
receiving, from an application of a first client device of a user, an indication of a request by the first device for a web page, the web page associated with a plurality of different variations of the web page;
determining an identifier associated with a user of the first client device;
applying a function to the identifier to generate an assignment identifier for the user;
wherein applying the function comprises:
generating a hash of the identifier using the function; and
normalizing the hash of the identifier to generate the assignment identifier that is within an interval, wherein the assignment identifier and other assignment identifiers associated with other users are distributed within the interval based on a proportion of corresponding users assigned to each of the plurality of different variations;
selecting a variation of the web page from the plurality of different variations of the web page that corresponds to the assignment identifier for the user; and
receiving, from an application of a second client device of the user that is a different client device type from the first client device of the user, an indication of a request by the second client device for the same web page requested by the first client device;
wherein the same selected variation of the web page is provided to both the first client device of the user and the second client device of the user.
13. The computer program product of claim 12 , wherein the function applied to the identifier is a deterministic, uniform and pseudorandom number generator.
14. The computer program product of claim 13 , wherein the code when executed by the computer further causes the computer to perform steps comprising:
determining an identifier associated with a variation test for the web page that includes the plurality of different variations of the web page; and
applying the deterministic, uniform and pseudorandomgenerator to a concatenation of the identifier and the identifier associated with the variation test for the web page to generate the normalized hash of the identifier.
15. The computer program product of claim 14 , wherein selecting the variation of the web page comprises:
identifying a policy for the web page that describes which variation of the web page to provide based on assignment identifiers;
comparing the normalized assignment identifier to the policy; and
selecting the variation of the web page to provide in response to receiving the identifier based on the comparison.
16. The computer program product of claim 15 , wherein the plurality of different variations of the web page comprise only a first variation of the web page and a second variation of the web page, and the policy includes a threshold indicative of whether to provide the first variation or the second variation in response to receiving the identifier, and wherein comparing the generated assignment identifier to the policy comprises:
comparing the assignment identifier to the threshold;
responsive to the assignment identifier being below the threshold, selecting the first variation of the web page to provide in response to receiving the identifier; and
responsive to the assignment identifier being above the threshold, selecting the second variation of the web page to provide in response to receiving the identifier.
17. The computer program product of claim 15 , wherein comparing the assignment identifier to the policy comprises:
identifying, from the policy, a mapping of a plurality of ranges of assignment identifiers to the plurality of variations of the web page, each range of assignment identifiers associated with a different variation of the web page;
identifying a range of normalized assignment identifiers from the plurality of ranges of assignment identifiers that includes the assignment identifier; and
identifying a variation of the web page that is associated with the identified range of assignment identifiers.
18. The computer program product of claim 12 , wherein providing the selected variation of the web page comprises:
transmitting one or more notifications to the content provider, the one or more notifications each including an indication of the selected variation of the web page to provide to the first client device and the second client device;
wherein the content provider provides the selected variation of the web page to the first client device and the second client device based on the notification.
19. A computer system of determining a variation of a web page to provide to client devices of a user, the computer system comprising:
a computer processor;
a non-transitory computer-readable storage medium storing executable code, the code when executed by the computer processor causes the computer processor to perform operations comprising:
receiving, from an application of a first client device of a user, an indication of a request by the first client device for a web page, the web page associated with a plurality of different variations of the web page;
determining an identifier associated with a user of the first client device;
applying a function to the identifier to generate an assignment identifier for the user;
wherein applying the function comprises:
generating a hash of the identifier using the function; and
normalizing the hash of the identifier to generate the assignment identifier that is within an interval, wherein the assignment identifier and other assignment identifiers associated with other users are distributed within the interval based on a proportion of corresponding users assigned to each of the plurality of different variations;
selecting a variation of the web page from the plurality of different variations of the web page that corresponds to the assignment identifier for the user; and
receiving, from an application of a second client device of the user that is a different client device type from the first client device of the user, an indication of a request by the second device from the same web page requested by the first client device;
wherein the same selected variation of the web page is provided to both the first client device of the user and the second client device of the user.