Automated model update pipeline

1 . A method, comprising:

executing, on a processor of a computing device, instructions that cause the computing device to perform operations, the operations comprising:

collecting user behavior data associated with content provided to users;

invoking an automatic model trainer of a trainer workflow system to train a new model to output a set of model parameters based upon a configuration specifying a target audience, features extracted from the user behavior data, and training model parameters;

in response to determining that the new model will outperform a deployed model on a content serving platform, invoking an automatic model updater to update the content serving platform with the new model and a ranking profile of the new model for serving content requests;

utilizing a multi-armed bandit algorithm to evaluate a plurality of models for content requests for transmitting traffic from a server to one or more remote computing devices;

configuring an online evaluation mechanism to periodically assign one or more scores to the plurality of models, including the new model and the deployed model, available to the content serving platform based upon the evaluations of performance of the plurality of models determined utilizing the multi-armed bandit algorithm;

responsive to collecting up-to-date user behavior data corresponding to a plurality of users, training at least some of the plurality of models on the up-to-date user behavior data corresponding to the plurality of users, evaluating performance of the trained models, and updating at least some scores for the trained models to generate updated scores, dynamically shifting traffic for serving the content requests amongst the plurality of models based upon the periodically assigned one or more scores associated with performance of the plurality of models;

configuring a continuous integration and continuous delivery pipeline to periodically scan and verify new models as the new models are generated and trained by the automatic model trainer of the trainer workflow system;

in response to the continuous integration and continuous delivery pipeline identifying a verified model, committing changes to a new branch within a remote repository of the content serving platform based upon the verified model; and

in response to the continuous integration and continuous delivery pipeline committing the changes to the new branch, raising a pull request to obtain review for the new branch.

2 . The method of claim 1 , comprising:

performing an offline evaluation of the new model to predict performance of the new model based upon evaluation metrics; and

compare the performance of the new model to performance of the deployed model to determine whether the new model out performs the deployed model.

3 . The method of claim 2 , wherein the evaluation metrics comprise at least one of precision, recall, area under curve, or normalized discounted cumulative gain.

4 . The method of claim 1 , comprising:

utilizing an online evaluation mechanism to score a plurality of models, including the new model and the deployed model, available to the content serving platform.

5 . The method of claim 4 , wherein the online evaluation mechanism utilizes a multi-armed bandit mechanism to score the plurality of models.

6 . The method of claim 5 , wherein the multi-armed bandit mechanism utilizes a Bayesian upper confidence bounds algorithm to select a selected model out of the plurality of models to route incoming traffic for serving the content requests.

7 . The method of claim 4 , comprising:

selecting a selected model out of the plurality of models for serving the content requests based upon the selected model having a score greater than scores of other models within the plurality of models.

8 . The method of claim 1 , comprising:

evaluating performance of the new model used to serve a first portion of incoming traffic for serving the content requests;

evaluating performance of the deployed model used to serve a second portion of incoming traffic for serving the content requests; and

adjusting at least one of the first portion or the second portion of the incoming traffic based upon the performance of the new model and the performance of the deployed model.

9 . The method of claim 1 , comprising:

configuring the automatic model trainer with training variables associated with filters for the user behavior data, a location to read the user behavior data, and a tuning range for hyperparameters.

10 . The method of claim 1 , comprising:

configuring the automatic model trainer with training variables associated with filters for the user behavior data.

11 . The method of claim 1 , comprising:

configuring the automatic model trainer with training variables associated with a location to read the user behavior data.

12 . The method of claim 1 , comprising:

configuring the automatic model trainer with training variables associated with a tuning range for hyperparameters.

13 . A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising:

training a new model to output a set of model parameters based upon a configuration specifying a target audience, features extracted from user behavior data, and training model parameters;

in response to determining that the new model will outperform a deployed model on a content serving platform, invoking an automatic model updater to update the content serving platform with the new model and a ranking profile of the new model for serving content requests;

utilizing a multi-armed bandit algorithm to evaluate a plurality of models for content requests for transmitting traffic from a server to one or more remote computing devices;

configuring an online evaluation mechanism to periodically assign one or more scores to the plurality of models, including the new model and the deployed model, available to the content serving platform based upon the evaluations of performance of the plurality of models determined utilizing the multi-armed bandit algorithm;

responsive to collecting up-to-date user behavior data corresponding to a plurality of users, training at least some of the plurality of models on the up-to-date user behavior data corresponding to the plurality of users, evaluating performance of the trained models, and updating at least some scores for the trained models to generate updated scores, dynamically shifting traffic for serving the content requests amongst the plurality of models based upon the periodically assigned one or more scores associated with performance of the plurality of models;

configuring a continuous integration and continuous delivery pipeline to periodically scan and verify new models as the new models are generated and trained by an automatic model trainer of a trainer workflow system;

in response to the continuous integration and continuous delivery pipeline identifying a verified model, committing changes to a new branch within a remote repository of the content serving platform based upon the verified model; and

in response to the continuous integration and continuous delivery pipeline committing the changes to the new branch, raising a pull request to obtain review for the new branch.

14 . The non-transitory machine readable medium of claim 13 , wherein the operations comprise:

utilizing the new model to serve exploitation traffic of a content request.

15 . The non-transitory machine readable medium of claim 13 , wherein the operations comprise:

ranking the performance of the models using a multi-armed bandit mechanism.

16 . The non-transitory machine readable medium of claim 13 , wherein the operations comprise:

utilizing the new model to serve exploration traffic of a content request.

17 . The non-transitory machine readable medium of claim 13 , wherein the operations comprise:

configuring an automatic model trainer to train new models using user behavior data, testing the new models, and releasing the new models for deployment for serving the content requests.

18 . The non-transitory machine readable medium of claim 13 , wherein the operations comprise:

configuring the automatic model trainer with training variables associated with at least one of filters for the user behavior data, a location to read the user behavior data, or a tuning range for hyperparameters.

19 . A computing device comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising:

training a new model to output a set of model parameters based upon a configuration specifying a target audience, features extracted from user behavior data, and training model parameters;

in response to determining that the new model will outperform a deployed model on a content serving platform, invoking an automatic model updater to update the content serving platform with the new model and a ranking profile of the new model for serving content requests;

utilizing a multi-armed bandit algorithm to evaluate a plurality of models for content requests for transmitting traffic from a server to one or more remote computing devices;

configuring an online evaluation mechanism to periodically assign one or more scores to the plurality of models, including the new model and the deployed model, available to the content serving platform based upon the evaluations of performance of the plurality of models determined utilizing the multi-armed bandit algorithm;

responsive to collecting up-to-date user behavior data corresponding to a plurality of users, training at least some of the plurality of models on the up-to-date user behavior data corresponding to the plurality of users, evaluating performance of the trained models, and updating at least some scores for the trained models to generate updated scores, dynamically shifting traffic for serving the content requests amongst the plurality of models based upon the periodically assigned one or more scores associated with performance of the plurality of models;

configuring a continuous integration and continuous delivery pipeline to periodically scan and verify new models as the new models are generated and trained by an automatic model trainer of a trainer workflow system;

in response to the continuous integration and continuous delivery pipeline identifying a verified model, committing changes to a new branch within a remote repository of the content serving platform based upon the verified model; and

in response to the continuous integration and continuous delivery pipeline committing the changes to the new branch, raising a pull request to obtain review for the new branch.

20 . The computing device of claim 19 , wherein the computing device periodically implements an automated process to collect up-to-date user behavior data, train new models on the up-to-date user behavior data, evaluate performance of the new models offline, update the new models and ranking profiles for the new models for serving content requests, and dynamically shift traffic for serving content requests amongst the plurality of models based upon performance of the plurality of models.