Granted Patent B2

US 10,585,778 · App. 15/682,163 · Granted Mar 10, 2020

Inventors: Leonid Boris Pekelis (San Bruno, CA); David Jonathan Max Walsh (San Francisco, CA); Ramesh Om Johari (San Francisco, CA)

Assignee: Optimizely, inc.

G06F11/3612G06F11/3688

Abstract

Claims (71)

1. A computer-implemented method of conducting an experiment, the method comprising:

receiving a plurality of indications of user interaction with a content item of a web page and variation of the content item that is distinct from the content item of the web page, wherein a plurality of client devices received either the content item or the variation of the content item from a content provider, wherein each indication specifies an occurrence or a nonoccurrence of a user conversion for the content item or the variation of the content item;

upon receiving a corresponding indication for each client device of the plurality of client devices interacting with the content item or the variation of the content item, calculating a difference in conversion rate of the user conversions between the content item and the variation of the content item based on the received plurality of indications of user conversions;

determining, after each of the user conversions, whether the difference in the conversion rate of the user conversions is within a confidence interval, wherein determining that the difference in the conversion rate is not within the confidence interval causes the experiment to reset;

receiving, from a client device, a request to view results of the experiment;

determining the results of the experiment based on a first conversion rate associated with user conversions of the content item and a second conversion rate associated with user conversions of the variation of the content item; and

providing the results of the experiment to the client device.

2. The computer-implemented method of claim 1 , wherein determining whether the difference in the conversion rate of the user conversions is within the confidence interval further comprises:

calculating the confidence interval indicative of a range of differences in conversion rate of user conversions between the content item and the variation of the content item;

responsive to receiving each of the plurality of the indications:

determining the difference in conversion rate of user conversions between the content item and the variation of the content item;

comparing the determined difference in conversion rate to the confidence interval;

responsive to the determined difference in conversion rate not being within the confidence interval, resetting the experiment; and

responsive to the determined difference in conversion rate being within the confidence interval, allowing the experiment to continue.

3. The computer-implemented method of claim 2 , wherein resetting the experiment comprises:

responsive to the content item being provided to a second plurality of client devices since the experiment was reset:

calculating an updated confidence interval based on user conversions with the content item and the variation of the content item that occurred after the experiment was reset; and

calculating an updated magnitude of a difference in conversion rate of user conversions between the content item and the variation of the content item that is required to deem the experiment conclusive after the reset.

4. The computer-implemented method of claim 3 , further comprising:

receiving indications that the content item and the variation of the content item are provided to the second plurality of client devices;

responsive to receiving each indication:

determining a difference in conversion rate of the user conversions between the content item and the variation of the content item that occurred after the experiment was reset; and

comparing the difference in conversion rate to the updated confidence interval; and

determining whether to reset the experiment based on the comparison.

5. The computer-implemented method of claim 1 , wherein the variation of the content item includes one or more modifications to elements of the content item.

6. The computer-implemented method of claim 5 , wherein the difference in the conversion rate of user conversions indicates performance of the content item with respect to the variation of the content item, and wherein the performance indicates increased conversions with the modified elements of the variation of the content item that are included in the variation of the content item.

7. The computer-implemented method of claim 6 , further comprising:

determining a magnitude of a difference in the conversion rate of user conversions between the content item and the variation of the content item that is required to deem the experiment conclusive.

8. The computer-implemented method of claim 1 , wherein the results of the experiment comprises at least one of a statistical significance of the experiment, a conversion rate associated with the content item and a conversion rate associated with the variation of the content item, a comparison between the conversion rate associated with the content item and the conversion rate associated with the variation of the content item, a number of users that received the content item and a number of users that received the variation of the content item, a total number of conversions associated with the content item and a total number of conversions associated with the variation of the content item, and a graph illustrating the conversion rate of the content item over time and the conversion rate of the variation of the content item over time.

9. The computer-implemented method of claim 8 , wherein the content item comprises one of a web page, a mobile device application, a video file, an audio file, a multimedia file, an image, a text document, or an advertisement.

10. The computer implemented method of claim 1 , wherein the results are determined without requiring a preset sample size of user conversions with the content item and the variation of the content item based on the calculated difference in conversion rate.

11. A computer program product comprising a non-transitory computer-readable storage medium including executable code to conduct an experiment, the code when executed by a computer processor causes the computer processor to perform steps comprising:

receiving a plurality of indications of user interaction with a content item and a variation of the content item that is distinct from the content item, wherein a plurality of client devices received either the content item or the variation of the content item from a content provider, wherein each indication specifies an occurrence or a nonoccurrence of a user conversion for the content item or the variation of the content item;

upon receiving each indication, calculating a difference in conversion rate of the user conversions between the content item and the variation of the content item based on the received plurality of indications of user conversions;

determining whether the difference in the conversion rate of the user conversions is within a confidence interval, wherein determining that the difference in the conversion rate is not within the confidence interval causes the experiment to reset;

receiving, from a client device, a request to view results of the experiment;

determining the results of the experiment based on a first conversion rate associated with user conversions of the content item and a second conversion rate associated with user conversions of the variation of the content item; and

providing the results of the experiment to the client device.

12. The computer program product of claim 11 , wherein to determine whether the difference in the conversion rate of the user conversions is within the confidence interval, the computer processor to perform further steps comprising:

calculating the confidence interval indicative of a range of differences in conversion rate of user conversions between the content item and the variation of the content item;

responsive to receiving each of the plurality of the indications:

determining the difference in conversion rate of user conversions between the content item and the variation of the content item;

comparing the determined difference in conversion rate to the confidence interval;

responsive to the determined difference in conversion rate not being within the confidence interval, resetting the experiment; and

responsive to the determined difference in conversion rate being within the confidence interval, allowing the experiment to continue.

13. The computer program product of claim 12 , wherein resetting the experiment comprises:

responsive to the content item being provided to a second plurality of client devices since the experiment was reset:

calculating an updated confidence interval based on user conversions with the content item and the variation of the content item that occurred after the experiment was reset; and

calculating an updated magnitude of a difference in conversion rate of user conversions between the content item and the variation of the content item that is required to deem the experiment conclusive after the reset.

14. The computer program product of claim 11 , wherein the variation of the content item includes one or more modifications to elements of the content item.

15. The computer program product of claim 14 , wherein the difference in conversion rate of user conversions indicates performance of the content item with respect to the variation of the content item, and wherein the performance indicates increased conversions with the modified elements of the variation of the content item that are included in the variation of the content item.

16. The computer program product of claim 15 , wherein the code when executed by the computer processor causes the computer processor to perform further steps comprising:

determining a magnitude of a difference in the conversion rate of user conversions between the content item and the variation of the content item that is required to deem the experiment conclusive.

17. A computer system to conduct an experiment, the computer system comprising:

a computer processor; and

a non-transitory computer-readable storage medium including executable code, the code when executed by the computer processor causes the computer processor to perform steps comprising:

receiving a plurality of indications of user interaction with a content item and at least a variation of the content item that is distinct from the content item, wherein a plurality of client devices received either the content item or the variation of the content item from a content provider, wherein each indication specifies an occurrence or a nonoccurrence of a user conversion for the content item of the variation of the content item;

upon receiving each indication, calculating a difference in conversion rate of the user conversions between the content item and the variation of the content item based on the received plurality of indications of user conversions;

determining whether the difference in the conversion rate of the user conversions is within a confidence interval, wherein determining that the difference in the conversion rate is not within the confidence interval causes the experiment to reset;

receiving, from a client device, a request to view results of the experiment;

determining the results of the experiment based on a first conversion rate associated with user conversions of the content item and a second conversion rate associated with user conversions of the variation of the content item; and

providing the results of the experiment to the client device.

18. The computer system of claim 17 , wherein to determine whether the difference in the conversion rate of the user conversions is within the confidence interval, the computer processor to perform further steps comprising:

calculating the confidence interval indicative of a range of differences in conversion rate of user conversions between the content item and the variation of the content item;

responsive to receiving each of the plurality of the indications:

determining the difference in conversion rate of user conversions between the content item and the variation of the content item;

comparing the determined difference in conversion rate to the confidence interval;

responsive to the determined difference in conversion rate not being within the confidence interval, resetting the experiment; and

responsive to the determined difference in conversion rate being within the confidence interval, allowing the experiment to continue.

19. The computer system of claim 17 , wherein the variation of the content item includes one or more modifications to elements of the content item.

20. The computer system of claim 19 , wherein the difference in conversion rate of user conversions indicates performance of the content item with respect to the variation of the content item, and wherein the performance indicates increased conversions with the modified elements of the variation of the content item that are included in the variation of the content item.

Assignments (10)

Continuity (2)