Experimentation in internet-connected applications and devices
A content variation experiment system for performing variation testing of web pages is disclosed. A content provider receives requests for a web page undergoing an experiment. The content provider determines a variation from a plurality of variations of the web page to provide to the user. The content provider makes the determination without sending a network request to an experiment definition system used to define the experiment thereby reducing network latency.
1. A method, comprising:
receiving, by a computer processor of a content provider system, information generated on an experiment definition system;
determining, by the computer processor of the content provider system, a variation of a content item from a plurality of different variations of the content item to provide to a client device without communicating with the experiment definition system, the determination based on an experiment definition file stored on the content provider system and the information generated on the experiment definition system, the information allocates users requesting the content item to one of the plurality of different variations of the content item;
including, by the computer processor of the content provider system, the information in the content item; and
providing, by the computer processor of the content provider system, the determined variation of the content item to the client device.
2. The method of claim 1 , wherein the content item is a web page.
3. The method of claim 1 , wherein the content item is a mobile application.
4. The method of claim 1 , wherein the content item is one or more of a video file, an audio file, an image file, text document, and an advertisement.
5. The method of claim 1 , further comprising:
receiving, from the experiment definition system, the experiment definition file; and
storing the experiment definition file received from the experiment definition system, the experiment definition file describing the attributes of the variation experiment for the content item.
6. The method of claim 5 ,
wherein the attributes include a definition for each of the plurality of different variations of the content item.
7. The method of claim 1 , further comprising:
tracking events associate with the content item; and
reporting the tracked events to the experiment definition system.
8. The method of claim 7 , further comprising;
transmitting a request for statistics associated with the variation experiment of the content item to the experiment definition system; and
receiving the statistics associated with the variation experiment of the content item from the experiment definition system, the statistics calculated based on the tracked events reported to the experiment definition system.
9. A non-transitory computer-readable storage medium storing executable code that when executed by a computer processor, cause the computer processor to:
receive information generated on an experiment definition system;
determine a variation of a content item from a plurality of different variations of the content item to provide to a client device without communicating with the experiment definition system, the determination based on an experiment definition file stored on a content provider system and the information generated on the experiment definition system, the information allocates users requesting the content item to one of the plurality of different variations of the content item;
include the information in the content item; and
provide the determined variation of the content item to the client device.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the content item is a web page.
11. The non-transitory computer-readable storage medium of claim 9 , wherein the content item is a mobile application.
12. The non-transitory computer-readable storage medium of claim 9 , wherein the computer processor further to:
receive, from the experiment definition system, the experiment definition file; and
store the experiment definition file received from the experiment definition system, the experiment definition file describing the attributes of the variation experiment for the content item.
13. The non-transitory computer-readable storage medium of claim 12 , wherein
the attributes include a definition for each of the plurality of different variations of the content item.
14. The non-transitory computer-readable storage medium of claim 9 , wherein the computer processor further to:
track events associate with the content item; and
report the tracked events to the experiment definition system.
15. The non-transitory computer-readable storage medium of claim 9 , wherein the computer processor further to:
transmit a request for statistics associated with the variation experiment of the content item to the experiment definition system; and
receive the statistics associated with the variation experiment of the content item from the experiment definition system, the statistics calculated based on the tracked events reported to the experiment definition system.
16. A content provider computer system, comprising:
a computer processor to:
receive information generated on an experiment definition system;
determine a variation of a content item from a plurality of different variations of the content item to provide to a client device without communicating with the experiment definition system, the determination based on an experiment definition file stored on the content provider computer system and information generated on the experiment definition system, the information allocates users requesting the content item to one of the plurality of different variations of the content item;
include the information in the content item; and
provide the determined variation of the content item to the client device.
17. The content provider computer system of claim 16 , wherein the content item is a web page.
18. The content provider computer system of claim 16 , wherein the content item is a mobile application.
19. The content provider computer system of claim 16 , wherein the computer processor further to:
receive, from the experiment definition system, the experiment definition file; and
store the experiment definition file received from the experiment definition system, the experiment definition file describing the attributes of the variation experiment for the content item.