Click-through rate model and generating customized copies using machine-learned large language models

1 . A method, comprising:

receiving a request to present one or more content items to a user for presentation;

retrieving at least a set of candidate content items for the user, wherein a candidate content item promotes a respective item;

obtaining one or more prediction scores for each candidate content item, wherein obtaining the one or more prediction scores for each candidate content item comprises:

providing prompts to a model serving system deploying a machine-learned model, the prompts comprising requests for the one or more prediction scores based on a natural language description of each candidate content item and the user, wherein the machine-learned model is configured as a transformer architecture including one or more attention layers, the transformer architecture coupled to receive a set of input tokens encoding the natural language description and generate a set of output tokens,

wherein the prompts request generation of a purchase score indicating a likelihood the user purchases the item, a relevance score indicating how relevant the item is to the user, and a click-through score indicating a likelihood the user will interact with the same candidate content item, and

receiving the prediction scores from the model serving system,

wherein the prediction scores include three or more prediction scores including the purchase score, the relevance score, and the click- through score;

extracting a set of features related to the user and the set of candidate content items;

applying a machine learned interaction model to the extracted set of features and the three or more prediction scores as input to the interaction model to generate interaction scores indicating likelihoods of interaction between the user and the candidate content items, wherein the three or more prediction scores are weighted differently from one another when the machine-learned interaction model is applied;

ranking the candidate content items according to the interaction scores, wherein the machine-learned interaction model is trained by:

obtaining at least a training instance including features related to content displayed to another user, and a relevance score, a prediction score, and a click-through score for the content, and

training parameters of the interaction model using the training instance;

selecting a content item based on the interaction scores; and

sending the selected content item to a user device for display to the user.

2 . The method of claim 1 , wherein retrieving at least the set of candidate content items for the user comprises:

obtaining item data describing a set of candidate items and user data describing the user,

providing prompts to the model serving system or another model serving system, the prompts comprising requests for one or more customized copies for each candidate item,

receiving the one or more customized copies, and

generating the set of candidate content items by incorporating each customized copy for a respective candidate item into content for the candidate item.

3 . The method of claim 1 , further comprising:

obtaining one or more training instances, a training instance corresponding to a content item another user interacted with or converted on, the training instance including a prompt and a known prediction score for the content item;

applying parameters of the machine-learned model to the prompt for the training instance to generate an estimated output; and

updating the parameters of the machine-learned model based on terms obtained from a loss function indicating a difference between the estimated output and the known prediction score.

4 . The method of claim 1 , further comprising:

receiving information regarding an item that is promoted in a content item campaign;

generating a prompt for input to the model serving system or another model serving system, the prompt specifying at least the item, an attribute, and historical information of the item;

parsing a response from the model serving system to extract a suggested attribute value for the attribute of the specified item;

populating form fields of a user interface (UI) based on the suggested attribute value; and

deploying the campaign to present one or more content items related to the campaign to users using the suggested attribute value.

5 . The method of claim 1 , wherein the candidate content item is at least one or a combination of an advertisement, a recommendation to purchase an item of a retailer, or a banner.

6 . A non-transitory computer-readable storage medium comprising stored instructions executable by a processor, the instructions when executed causing the processor to:

receive a request to present one or more content items to a user for presentation;

retrieve at least a set of candidate content items for the user, wherein a candidate content item promotes a respective item;

obtain one or more prediction scores for each candidate content item, wherein obtaining the one or more prediction scores for each candidate content item comprises:

providing prompts to a model serving system deploying a machine-learned model, the prompts comprising requests for the one or more prediction scores based on a natural language description of each candidate content item and the user, wherein the machine-learned model is configured as a transformer architecture including one or more attention layers, the transformer architecture coupled to receive a set of input tokens encoding the natural language description and generate a set of output tokens,

wherein the prompts request generation of a purchase score indicating a likelihood the user purchases the item, a relevance score indicating how relevant the item is to the user, and a click-through score indicating a likelihood the user will interact with the same candidate content item, and

receiving the prediction scores from the model serving system, wherein the prediction scores include three or more prediction scores including the purchase score, the relevance score, and the click-through score;

extract a set of features related to the user and the set of candidate content items;

apply a machine learned interaction model to the extracted set of features and the three of more prediction scores as input to the interaction model to generate interaction scores indicating likelihoods of interaction between the user and the candidate content items, wherein the three or more prediction scores are weighted differently from one another when the machine-learned interaction model is applied;

rank the candidate content items according to the interaction scores, wherein the machine-learned interaction model is trained by:

obtaining at least a training instance including features related to content displayed to another user, and a relevance score, a prediction score, and a click-through score for the content, and

training parameters of the interaction model using the training instance; and

select a content item based on the interaction scores; and

send the selected content item to a user device for display to the user.

7 . The non-transitory computer-readable storage medium of claim 6 , further comprising instructions when executed causing the processor to:

obtain item data describing a set of candidate items and user data describing the user,

provide prompts to the model serving system or another model serving system, the prompts comprising requests for one or more customized copies for each candidate item,

receive the one or more customized copies, and

generate the set of candidate content items by incorporating each customized copy for a respective candidate item into content for the candidate item.

8 . The non-transitory computer-readable storage medium of claim 6 , further comprising instructions when executed causing the processor to:

obtain one or more training instances, a training instance corresponding to a content item another user interacted with or converted on, the training instance including a prompt and a known prediction score for the content item;

apply parameters of the machine-learned model to the prompt for the training instance to generate an estimated output; and

update the parameters of the machine-learned model based on terms obtained from a loss function indicating a difference between the estimated output and the known prediction score.

9 . The non-transitory computer-readable storage medium of claim 6 , further comprising instructions when executed causing the processor to:

receive information regarding an item that is promoted in a content item campaign;

generate a prompt for input to the model serving system or another model serving system, the prompt specifying at least the item, an attribute, and historical information of the item;

parse a response from the model serving system to extract a suggested attribute value for the attribute of the specified item;

populate form fields of a user interface (UI) based on the suggested attribute value; and

deploy the campaign to present one or more content items related to the campaign to users using the suggested attribute value.

10 . The non-transitory computer-readable storage medium of claim 6 , wherein the candidate content item is at least one or a combination of an advertisement, a recommendation to purchase an item of a retailer, or a banner.

11 . A computer system comprising:

a processor; and

a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the computer system to:

receive a request to present one or more content items to a user for presentation;

retrieve at least a set of candidate content items for the user, wherein a candidate content item promotes a respective item;

obtain one or more prediction scores for each candidate content item, wherein obtaining the one or more prediction scores for each candidate content item comprises:

providing prompts to a model serving system deploying a machine-learned model, the prompts comprising requests for the one or more prediction scores based on a natural language description of each candidate content item and the user, wherein the machine-learned model is configured as a transformer architecture including one or more attention layers, the transformer architecture coupled to receive a set of input tokens encoding the natural language description and generate a set of output tokens,

wherein the prompts request generation of a purchase score indicating a likelihood the user purchases the item, a relevance score indicating how relevant the item is to the user, and a click-through score indicating a likelihood the user will interact with the same candidate content item, and

receiving the prediction scores from the model serving system, wherein the prediction scores include three or more prediction scores including the purchase score, the relevance score, and the click-through score;

extract a set of features related to the user and the set of candidate content items;

apply a machine learned interaction model to the extracted set of features and the three or more prediction scores as input to the interaction model to generate interaction scores indicating likelihoods of interaction between the user and the candidate content items, wherein the three or more prediction scores are weighted differently from one another when the machine-learned model is applied;

rank the candidate content items according to the interaction scores, wherein the machine-learned interaction model is trained by:

obtaining at least a training instance including features related to content displayed to another user, and a relevance score, a prediction score, and a click-through score for the content, and

training parameters of the interaction model using the training instance;

select a content item based on the interaction scores; and

send the selected content item to a user device for display to the user.

12 . The computer system of claim 11 , further comprising instructions when executed causing the computer system to:

obtain item data describing a set of candidate items and user data describing the user;

provide prompts to the model serving system or another model serving system, the prompts comprising requests for one or more customized copies for each candidate item;

receive the one or more customized copies; and

generate the set of candidate content items by incorporating each customized copy for a respective candidate item into content for the candidate item.