Granted Patent B2

US 11,551,281 · App. 16/775,067 · Granted Jan 10, 2023

Inventor: Ethan Berl (San Francisco, CA)

Assignee: OPTIMIZELY, INC.

G06Q30/0631G06N20/00

Abstract

Claims (79)

1. A method comprising:

receiving, at a processing device of an experiment system, a parameter associated with a set of recommendation algorithms, the set of recommendation algorithms operating as an experiment for content hosted by a content provider;

generating, by the processing device, a first combination of the recommendation algorithms having a first set of weights, the first combination comprising a first subset of the set of recommendation algorithms;

assigning each of the first combination of the recommendation algorithms to a respective one of the first set of weights;

generating, by the processing device, a second combination of the recommendation algorithms having a second set of weights, the second combination comprising a second subset of the set of recommendation algorithms;

assigning each of the second combination of the recommendation algorithms to a respective one of the second set of weights, wherein at least one weight of the second set of weights is different from the first set of weights;

generating, by the processing device, first content recommendations for a first plurality of users using the first combination of the recommendation algorithms and the associated first set of weights;

generating second content recommendations for a second plurality of users, different from the first plurality of users, using the second combination of the recommendation algorithms and the associated second set of weights;

comparing, by the processing device, a first performance of the first content recommendations with respect to the parameter to a performance of the second content recommendations with respect to the parameter; and

allocating, by a processing device, a portion of new users to the first combination of the recommendation algorithms responsive to determining that the first content recommendations generated using the first combination of the recommendation algorithms result in a better performance with respect to the parameter than the second content recommendations generated using the second combination of the recommendation algorithms, wherein subsequent content recommendations generated by the first combination of the recommendation algorithms are integrated into a user interface through which the portion of new users interact with the content provider, and

wherein the respective algorithms of the first combination of the recommendation algorithms are the same as the respective algorithms of the second combination of the recommendation algorithms.

2. The method of claim 1 , wherein generating the first content recommendations comprises:

generating an initial recommendation using each of the recommendation algorithms in the first combination of the recommendation algorithms;

multiplying each of the initial recommendations by the weight assigned to the recommendation algorithm that generated the initial recommendation;

normalizing the initial recommendations based on a strength score of each of the initial recommendations; and

combining the initial recommendations together to generate the first content recommendations.

3. The method of claim 1 , wherein the first combination of the recommendation algorithms is generated using a machine learning algorithm.

4. The method of claim 1 , further comprising:

in response to a new user, determining whether to generate content recommendations for the new user using the first combination of the recommendation algorithms or a new combination of the recommendation algorithms;

in response to determining that a new combination of the recommendation algorithms should be used, generating the new combination of the recommendation algorithms and assigning a weight to each algorithm in the new combination of the recommendation algorithms; and

generating a content recommendation for the new user using the new combination.

5. The method of claim 4 , wherein generating the new combination of the recommendation algorithms and assigning the weight to each algorithm in the new combination of the recommendation algorithms comprises:

selecting the respective weight for each algorithm of the new combination of the recommendation algorithm based on a proximity of the respective weight to the first set of weights and the second set of weights.

6. The method of claim 1 , further comprising:

for each of the first and second combinations of the recommendation algorithms, monitoring a time variation of the performance of the combination with respect to the parameter.

7. The method of claim 1 , wherein one or more of the first combination of the recommendation algorithms is an algorithm hosted separately from the remaining recommendation algorithms of the first combination of the recommendation algorithms.

8. A system comprising:

a memory to store a set of recommendation algorithms; and

a processing device operatively coupled to the memory, the processing device to:

receive a parameter associated with the set of recommendation algorithms, the set of recommendation algorithms operating as an experiment for content hosted by a content provider;

generate a first combination of the recommendation algorithms having a first set of weights, the first combination comprising a first subset of the set of recommendation algorithms;

assign each of the first combination of the recommendation algorithms to a respective one of the first set of weights;

generate a second combination of the recommendation algorithms having a second set of weights, the second combination comprising a second subset of the set of recommendation algorithms;

assign each of the second combination of the recommendation algorithms to a respective one of the second set of weights, wherein at least one weight of the second set of weights is different from the first set of weights;

generate first content recommendations for a first plurality of users using the first combination of the recommendation algorithms and the associated first set of weights;

generate second content recommendations for a second plurality of users, different from the first plurality of users, using the second combination of the recommendation algorithms and the associated second set of weights;

compare a first performance of the first content recommendations with respect to the parameter to a performance of the second content recommendations with respect to the parameter; and

allocate a portion of new users to the first combination of the recommendation algorithms responsive to determining that the first content recommendations generated using the first combination of the recommendation algorithms result in a better performance with respect to the parameter than the second content recommendations generated using the second combination of the recommendation algorithms, wherein subsequent content recommendations generated by the first combination of the recommendation algorithms are integrated into a user interface through which the portion of new users interact with the content provider,

wherein the respective algorithms of the first combination of the recommendation algorithms are the same as the respective algorithms of the second combination of the recommendation algorithms.

9. The system of claim 8 , wherein to generate the first content recommendations, the processing device is to:

generate an initial recommendation using each of the recommendation algorithms in the first combination of the recommendation algorithms;

multiply each of the initial recommendations by the weight assigned to the recommendation algorithm that generated the initial recommendation;

normalize the set of initial recommendations based on a strength score of each of the initial recommendations; and

combine the initial recommendations together to generate the first content recommendations.

10. The system of claim 8 , wherein the processing device uses a machine learning algorithm to generate the first combination of the recommendation algorithms.

11. The system of claim 8 , wherein the processing device is further to:

in response to a new user, determine whether to generate content recommendations for the new user using the first combination of the recommendation algorithms or a new combination of the recommendation algorithms;

in response to determining that a new combination of the recommendation algorithms should be used, generate the new combination of the recommendation algorithms and assign a weight to each algorithm in the new combination of the recommendation algorithms; and

generate a content recommendation for the new user using the new combination.

12. The system of claim 11 , wherein to generate the new combination of the recommendation algorithms and assign the weight to each algorithm in the new combination of the recommendation algorithm, the processing device is to:

select the respective weight for each algorithm of the new combination of the recommendation algorithm based on a proximity of the respective weight to the first set of weights and the second set of weights.

13. The system of claim 8 , wherein the processing device is further to:

for each of the first and second combinations of the recommendation algorithms, monitor a time variation of the performance of the combination with respect to the parameter.

14. The system of claim 8 , wherein one or more of the first combination of the recommendation algorithms is an algorithm hosted separately from the remaining recommendation algorithms of the first combination of the recommendation algorithms.

15. A non-transitory computer readable medium, having instructions stored thereon which, when executed by a processing device, cause the processing device to:

receive a parameter associated with a set of recommendation algorithms, the set of recommendation algorithms operating as an experiment for content hosted by a content provider;

generate, by the processing device, a first combination of the recommendation algorithms having a first set of weights, the first combination comprising a first subset of the set of recommendation algorithms;

assign each of the first combination of the recommendation algorithms to a respective one of the first set of weights;

generate, by the processing device, a second combination of the recommendation algorithms having a second set of weights, the second combination comprising a second subset of the set of recommendation algorithms;

assign each of the second combination of the recommendation algorithms to a respective one of the second set of weights, wherein at least one weight of the second set of weights is different from the first set of weights;

generate, by the processing device, first content recommendations for a first plurality of users using the first combination of the recommendation algorithms and the associated first set of weights;

generate second content recommendations for a second plurality of users, different from the first plurality of users, using the second combination of the recommendation algorithms and the associated second set of weights;

compare a first performance of the first content recommendations with respect to the parameter to a performance of the second content recommendations with respect to the parameter; and

allocate a portion of new users to the first combination of the recommendation algorithms responsive to determining that the first content recommendations generated using the first combination of the recommendation algorithms result in a better performance with respect to the parameter than the second content recommendations generated using the second combination of the recommendation algorithms, wherein subsequent content recommendations generated by the first combination of the recommendation algorithms are integrated into a user interface through which the portion of new users interact with the content provider,

wherein the respective algorithms of the first combination of the recommendation algorithms are the same as the respective algorithms of the second combination of the recommendation algorithms.

16. The non-transitory computer readable medium of claim 15 , wherein to generate the first content recommendations, the processing device is to:

generate an initial recommendation using each of the recommendation algorithms in the first combination of the recommendation algorithms;

multiply each of the initial recommendations by the weight assigned to the recommendation algorithm that generated the initial recommendation;

normalize the set of initial recommendations based on a strength score of each of the initial recommendations; and

combine the initial recommendations together to generate the first content recommendations.

17. The non-transitory computer readable medium of claim 15 , wherein the processing device uses a machine learning algorithm to generate the first combination of the recommendation algorithms.

18. The non-transitory computer readable medium of claim 15 , wherein the processing device is further to:

in response to a new user, determine whether to generate content recommendations for the new user using the first combination of the recommendation algorithms or a new combination of the recommendation algorithms;

in response to determining that a new combination of the recommendation algorithms should be used, generate the new combination of the recommendation algorithms and assign a weight to each algorithm in the new combination of the recommendation algorithms; and

generate a content recommendation for the new user using the new combination.

19. The non-transitory computer readable medium of claim 18 , wherein to generate the new combination of the recommendation algorithms and assign the weight to each algorithm in the new combination of the recommendation algorithm, the processing device is to:

select the respective weight for each algorithm of the new combination of the recommendation algorithm based on a proximity of the respective weight to the first set of weights and the second set of weights.

20. The non-transitory computer readable medium of claim 15 , wherein the processing device is further to:

for each of the first and second combinations of the recommendation algorithms, monitor a time variation of the performance of the combination with respect to the parameter.

Assignments (10)