Model for serving exploration traffic

1. A method, comprising:

executing, on a processor of a computing device, instructions that cause the computing device to perform operations, the operations comprising:

determining an amount of spend over a timespan by a content provider to provide content items of the content provider through a content serving platform to client devices of users, wherein the content items of the content provider are available for serving non-exploration traffic from client devices in a deterministic manner using a user engagement model;

training the user engagement model to predict one or more likelihoods of users engaging with an exploration content item;

determining a confidence metric indicative of a confidence of the user engagement model in association with predicting a likelihood of users engaging with the exploration content item;

based upon the confidence metric exceeding a threshold, removing the exploration content item from an exploration bucket and adding the exploration content item into a non-exploration bucket as a content item to serve using the user engagement model;

determining a number of exploration impressions of users viewing exploration content items of the content provider over the timespan, wherein the exploration content items of the content provider are available for serving exploration traffic from the client devices in a non-deterministic manner;

determining a return on exploration impression (ROEI) metric for the content provider based upon a ratio of (i) the amount of spend to provide the content items available for serving the non-exploration traffic to (ii) the number of exploration impressions of the users viewing the exploration content items available for serving the exploration traffic;

utilizing the ROEI metric to rank available exploration content items of content providers for serving the exploration traffic; and

serving the exploration traffic using exploration content items selected based upon percentages of exploration traffic assigned to the content providers based upon ROEI metrics for the content providers.

2. The method of claim 1 , comprising:

determining and enforcing a maximum number of simultaneous content items of the content provider that can be maintained for serving the exploration traffic based upon the ROEI metric and a configurable base value.

3. The method of claim 1 , comprising:

determining a percentage of exploration traffic for the content provider based upon a minimum exploration percentage metric.

4. The method of claim 1 , comprising:

generating an exploration model for the content serving platform to use for selecting exploration content items to serve for the exploration traffic.

5. The method of claim 4 , wherein the generating an exploration model comprises:

utilizing the ROEI metrics to populate the exploration model with percentages of exploration traffic to serve using exploration content items of the content providers.

6. The method of claim 4 , comprising:

populating the exploration model with a first entry for a first content provider, wherein the first entry comprises a first identifier of the first content provider, a first ROEI metric for the first content provider, and a first maximum number of simultaneous exploration content items for the first content provider.

7. The method of claim 6 , comprising:

populating the exploration model with a second entry for a second content provider, wherein the second entry comprises a second identifier of the second content provider, a second ROEI metric for the second content provider, and a second maximum number of simultaneous exploration content items for the second content provider.

8. The method of claim 1 , wherein the determining an amount of spend comprises:

determining the amount of spend based upon an amount of non-exploration spend over the timespan by the content provider.

9. The method of claim 4 , comprising:

populating the exploration model with an entry for the exploration content item.

10. A computing device comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising:

determining an amount of spend over a timespan by a content provider to provide content items of the content provider through a content serving platform to client devices of users, wherein the content items of the content provider are used to serve non-exploration traffic;

training a user engagement model to predict one or more likelihoods of users engaging with an exploration content item;

determining a confidence metric indicative of a confidence of the user engagement model in association with predicting a likelihood of users engaging with the exploration content item;

based upon the confidence metric exceeding a threshold, removing the exploration content item from an exploration bucket and adding the exploration content item into a non-exploration bucket as a content item to serve using the user engagement model;

determining a number of exploration impressions of users viewing exploration content items of the content provider over the timespan, wherein the exploration content items of the content provider are used to serve exploration traffic;

determining a return on exploration impression (ROEI) metric for the content provider based upon a ratio of (i) the amount of spend to provide the content items used to serve the non-exploration traffic to (ii) the number of exploration impressions of the users viewing the exploration content items used to serve the exploration traffic;

utilizing the ROEI metric to rank available exploration content items of content providers for serving the exploration traffic; and

serving the exploration traffic using exploration content items selected based upon percentages of exploration traffic assigned to the content providers based upon ROEI metrics for the content providers.

11. The computing device of claim 10 , comprising:

generating an exploration model for the content serving platform to use for selecting exploration content items to serve for exploration traffic; and

periodically updating the exploration model.

12. The computing device of claim 10 , comprising:

generating an exploration model for the content serving platform to use for selecting exploration content items to serve for exploration traffic.

13. The computing device of claim 12 , comprising:

populating the exploration model with an entry specifying a first percentage of exploration traffic for a first exploration content item.

14. The computing device of claim 13 , comprising:

adjusting the percentage of exploration traffic based upon a minimum exploration percentage metric.

15. A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising:

determining an amount of spend over a timespan by a content provider to provide content items of the content provider through a content serving platform to client devices of users, wherein the content items of the content provider are available for serving non-exploration traffic;

training a user engagement model to predict one or more likelihoods of users engaging with an exploration content item;

determining a confidence metric indicative of a confidence of the user engagement model in association with predicting a likelihood of users engaging with the exploration content item;

based upon the confidence metric exceeding a threshold, removing the exploration content item from an exploration bucket and adding the exploration content item into a non-exploration bucket as a content item to serve using the user engagement model;

determining a number of exploration impressions of users viewing exploration content items of the content provider over the timespan, wherein the exploration content items of the content provider are available for serving exploration traffic;

determining a return on exploration impression (ROEI) metric corresponding to a ratio of (i) the amount of spend to provide the content items available for serving the non-exploration traffic to (ii) the number of exploration impressions of the users viewing the exploration content items available for serving the exploration traffic;

utilizing the ROEI metric to rank available exploration content items of content providers for serving the exploration traffic; and

serving the exploration traffic using exploration content items selected based upon percentages of exploration traffic assigned to the content providers based upon ROEI metrics for the content providers.

16. The non-transitory machine readable medium of claim 15 , wherein the operations comprise:

determining and enforcing a maximum number of simultaneous content items of the content provider that can be maintained based upon the ROEI metric and a configurable base value.

17. The non-transitory machine readable medium of claim 16 , wherein the operations comprise:

determining the configurable base value based upon a minimum percentage metric.

18. The non-transitory machine readable medium of claim 15 , wherein the operations comprise:

generating an exploration model for the content serving platform to use for selecting exploration content items to serve for the exploration traffic.

19. The non-transitory machine readable medium of claim 18 , wherein the generating an exploration model comprises:

utilizing the ROEI metrics to populate the exploration model with percentages of exploration traffic to serve using exploration content items of the content providers.

20. The non-transitory machine readable medium of claim 18 , wherein the operations comprise:

populating the exploration model with a first entry for a first content provider; and

populating the exploration model with a second entry for a second content provider.