Granted Patent B2

US 11,397,785 · App. 16/590,050 · Granted Jul 26, 2022

Inventors: Dan Siroker (Palo Alto, CA); Whelan Jefferson Boyd (New York City, NY); Hao Xia (San Francisco, CA); Zachary Leach (San Francisco, CA)

Assignee: Optimizely, Inc.

G06F16/9535G06Q30/0271G06Q30/0277H04L67/22

Abstract

Claims (56)

1. A method comprising:

receiving, at an experiment system from a client device that is separate from the experiment system, a browsing history associated with a user of the client device, wherein the browsing history is generated by a client-side experimentation agent associated with a control version web page, wherein the client-side experimentation agent is provided to the client device by a content provider system to execute on the client device, and wherein the content provider system is to operate in conjunction with the experiment system to perform variation testing on the control version web page at the client device, wherein the variation testing measures performance of the control version web page against one or more modifications to the control version web page;

receiving, at the experiment system from the content provider system, one or more keywords associated with a target audience for a first variant of the control version web page, the first variant of the control version web page comprising a first modification of positions within the control version web page of one or more items of content of the control version web page, wherein the first modification is based on the target audience, and wherein the first modification of the positions within the control version web page of the one or more items of content of the control version web page alters a first position of a first portion of the control version web page while leaving a second position of a second portion of the control version web page unchanged;

determining, by a processing device of the experiment system, a first score for the user of the client device based on the received browsing history for the user and the one or more keywords associated with the target audience;

comparing, by the processing device of the experiment system, the first score to a score threshold to determine whether the user is within the target audience;

causing the first variant of the control version web page to be displayed at the client device associated with the user, wherein the client-side experimentation agent receives a command to download and apply the first modification to the positions within the control version web page of the one or more items of content of the control version web page;

receiving a subsequent browsing history for the user;

determining a second score for the user based on the subsequent browsing history

determining whether the second score for the user satisfies the score threshold; and

in response to determining that the second score for the user does not satisfy the score threshold, causing a second variant of the control version web page to be presented at the client device of the user, wherein the second variant of the control version web page comprises a second modification of the control version web page that alters the first position of the first portion of the control version web page to a third position while leaving the second position of the second portion of the control version web page unchanged.

2. The method of claim 1 , further comprising:

receiving, from the content provider system, the score threshold for the target audience.

3. The method of claim 1 , further comprising:

receiving an indication of one or more content items of a web page consumed by the user, the indication causing the one or more content items to be used to determine the first score for the user.

4. The method of claim 1 , wherein the browsing history comprises one or more content items consumed by the user, the method further comprising:

determining text associated with the one or more content items, wherein determining the text comprises at least one of identifying text presented by the one or more content items, performing image recognition on the one or more content items or performing audio recognition on the one or more content items.

5. The method of claim 4 , further comprising:

utilizing word embeddings to assign a corresponding vector having an associated directionality to each of the one or more content items, wherein each of the word embeddings represents a respective proximity of each of the one or more content items to the one or more keywords associated with the target audience.

6. A system comprising:

a memory; and

a processing device, operatively coupled to the memory, to:

receive, at an experiment system from a client device that is separate from the experiment system, a browsing history associated with a user of the client device, wherein the browsing history is generated by a client-side experimentation agent associated with a control version web page, wherein the client-side experimentation agent is provided to the client device by a content provider system to execute on the client device, and wherein the content provider system is to operate in conjunction with the experiment system to perform variation testing on the control version web page at the client device, wherein the variation testing measures performance of the control version web page against one or more modifications to the control version web page;

receive, at the experiment system from the content provider system, one or more keywords associated with a target audience for a first variant of the control version web page, the first variant of the control version web page comprising a first modification of positions within the control version web page of one or more items of content of the control version web page, wherein the first modification is based on the target audience, and wherein the first modification of the positions within the control version web page of the one or more items of content of the control version web page alters a first position of a first portion of the control version web page while leaving a second position of a second portion of the control version web page unchanged;

determine, at the experiment system, a first score for the user of the client device based on the received browsing history for the user and the one or more keywords associated with the target audience;

compare, at the experiment system, the first score to a score threshold to determine whether the user is within the target audience;

cause the first variant of the control version web page to be displayed at the client device associated with the user, wherein the client-side experimentation agent receives a command to download and apply the first modification to the positions within the control version web page of the one or more items of content of the control version web page;

receive a subsequent browsing history for the user;

determine a second score for the user based on the subsequent browsing history

determine whether the second score for the user satisfies the score threshold; and

in response to determining that the second score for the user does not satisfy the score threshold, cause a second variant of the control version web page to be presented at the client device of the user, wherein the second variant of the control version web page comprises a second modification of the control version web page that alters the first position of the first portion of the control version web page to a third position while leaving the second position of the second portion of the control version web page unchanged.

7. The system of claim 6 , wherein the processing device is further to:

receive, from the content provider system, the score threshold for the target audience.

8. The system of claim 6 , wherein the processing device is further to:

receive an indication of one or more content items of a web page consumed by the user, the indication causing the one or more content items to be used to determine the first score of the user.

9. The system of claim 6 , wherein the browsing history comprises one or more content items consumed by the user, and

wherein the processing device is further to:

determine text associated with the one or more content items, wherein to determine the text, the processing device is to perform at least one of identifying text presented by the one or more content items, performing image recognition on the one or more content items or performing audio recognition on the one or more content items.

10. The system of claim 9 , wherein the processing device is further to:

utilize word embeddings to assign a corresponding vector having an associated directionality to each of the one or more content items, wherein each of the word embeddings represents a respective proximity of each of the one or more content items to the one or more keywords associated with the target audience.

11. A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:

receive, at an experiment system from a client device that is separate from the experiment system, a browsing history associated with a user of the client device, wherein the browsing history is generated by a client-side experimentation agent associated with a control version web page, wherein the client-side experimentation agent is provided to the client device by a content provider system to execute on the client device, and wherein the content provider system is to operate in conjunction with the experiment system to perform variation testing on the control version web page at the client device, wherein the variation testing measures performance of the control version web page against one or more modifications to the control version web page;

receive, at the experiment system from the content provider system, one or more keywords associated with a target audience for a first variant of the control version web page, the first variant of the control version web page comprising a first modification of positions within the control version web page of one or more items of content of the control version web page, wherein the first modification is based on the target audience, and wherein the first modification of the positions within the control version web page of the one or more items of content of the control version web page alters a first position of a first portion of the control version web page while leaving a second position of a second portion of the control version web page unchanged;

determine, at the experiment system, a first score for the user of the client device based on the received browsing history for the user and the one or more keywords associated with the target audience;

compare, by the processing device of the experiment system, the first score to a score threshold to determine whether the user is within the target audience;

cause the first variant of the control version web page to be displayed at the client device associated with the user, wherein the client-side experimentation agent receives a command to download and apply the first modification to the positions within the control version web page of the one or more items of content of the control version web page;

receive a subsequent browsing history for the user;

determine a second score for the user based on the subsequent browsing history

determine whether the second score for the user satisfies the score threshold; and

in response to determining that the second score for the user does not satisfy the score threshold, cause a second variant of the control version web page to be presented at the client device of the user, wherein the second variant of the control version web page comprises a second modification of the control version web page that alters the first position of the first portion of the control version web page to a third position while leaving the second position of the second portion of the control version web page unchanged.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the processing device is further to:

receive, from the content provider system, the score threshold for the target audience.

13. The non-transitory computer-readable storage medium of claim 11 , wherein the processing device is further to:

receive an indication of one or more content items of a web page consumed by the user, the indication causing the one or more content items to be used to determine the first score for the user.

14. The non-transitory computer-readable storage medium of claim 11 , wherein the browsing history comprises one or more content items consumed by the user, and

wherein the processing device is further to:

determine text associated with the one or more content items, wherein to determine the text, the processing device is to perform at least one of identifying text presented by the one or more content items, performing image recognition on the one or more content items or performing audio recognition on the one or more content items.

Assignments (10)