Click-through prediction for targeted content

1. A method of selecting a first candidate message from a set of candidate messages for display at a client device based on a probability, as determined by at least two learning models, that a user of the client device will select the first candidate message, the method comprising:

receiving, by a computing device and from the client device of the user, a request for one or more candidate messages from the set of candidate messages to display at the client device together with a set of messages associated with the user in a social network messaging service;

calculating, by the computing device, using a machine learning model that is based on a point-wise learning model and a pair-wise learning model, a candidate score associated with the probability that the user will select the first candidate message from the set of candidate messages, wherein:

the point-wise learning model calculates a corresponding first candidate score for each candidate message of the set of candidate messages based on a first likelihood that the user of the client device will select that candidate message, and generates a ranking of the set of candidate messages based on the corresponding first candidate score for each candidate message;

the pair-wise learning model adjusts the first candidate score of at least one candidate message of the set of candidate messages based on a second likelihood that the user will select the at least one candidate message instead of another candidate message of the set of candidate messages, and reorders the ranking of the set of candidate messages based on the adjusted first candidate score of the at least one candidate message; and

the candidate score for the first candidate message is based on the reordered ranking of the set of candidate messages;

determining, by the computing device, that the candidate score satisfies a threshold score;

sending, by the computing device and for display at the client device, the first candidate message;

receiving, by the computing device and from the client device, an impression callback time that indicates a time when the first candidate message was displayed at the client device;

receiving, by the computing device and from the client device, an engagement callback time that indicates a time when the user selected the first candidate message at the client device; and

training, by the computing device, the machine learning model based at least in part on a difference between the impression callback time and the engagement callback time.

2. The method of claim 1 , wherein training the machine learning model further comprises:

training, by the computing device, the machine learning model based on training instances for single candidate messages and training instances for pairs of candidate messages.

3. The method of claim 2 , further comprising:

receiving, by the computing device and from a plurality of client devices, a continuous stream of training instances that correspond to the candidate message displayed at the plurality of client devices, wherein each training instance indicates at least one of the impression callback or the engagement callback associated with the candidate message of the set of candidate messages; and

wherein training the machine learning model comprises training the machine learning model based at least in part on the continuous stream of training instances that correspond to the candidate message.

4. The method of claim 1 , wherein determining the candidate score comprises:

determining, by the computing device and based at least in part on the probability that the user will select the candidate message and on a bid price that an advertiser will pay if the user selects the candidate message, the candidate score for the candidate message.

5. The method of claim 1 , wherein sending the candidate message comprises:

sending, by the computing device, the candidate message in a set of candidate messages, wherein each respective candidate message in the set of candidate messages has a respective candidate score that satisfies the threshold score.

6. A computing device for selecting a first candidate message from a set of candidate messages for display at a client device based on a probability, as determined by at least two learning models, that a user of the client device will select the first candidate message, the computing device comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium storing instructions that are executable by the at least one processor to:

receive, from the client device of the user, a request for one or more candidate messages from the set of candidate messages to display at the client device together with a set of messages, wherein the set of messages is associated with the user in a social network messaging service;

calculate, using a machine learning model that is based on a point-wise learning model and a pair-wise learning model, a candidate score associated with the probability that the user will select the first candidate message from the set of candidate messages, wherein:

the point-wise learning model calculates a corresponding first candidate score for each candidate message of the set of candidate messages based on a first likelihood that the user of the client device will select that candidate message, and generates a ranking of the set of candidate messages based on the corresponding first candidate score for each candidate message;

the pair-wise learning model adjusts the first candidate score of at least one candidate message of the set of candidate messages based on a second likelihood that the user will select the at least one candidate message instead of another candidate message of the set of candidate messages, and reorders the ranking of the set of candidate messages based on the adjusted first candidate score of the at least one candidate message; and

the candidate score for the first candidate message is based on the reordered ranking of the set of candidate messages;

determine that the candidate score satisfies a threshold score;

send, for display at the client device, the first candidate message;

receive, from the client device, an impression callback time that indicates a time when the first candidate message was displayed at the client device;

receive, from the client device, an engagement callback time that indicates a time when the user selected the first candidate message at the client device; and

train the machine learning model based at least in part on a difference between the impression callback time and the engagement callback time.

7. The computing device of claim 6 , wherein the instructions that are executable by the at least one processor to train the machine learning model comprise instructions that are executable by the at least one processor to:

train the machine learning model based on training instances for single candidate messages and training instances for pairs of candidate messages.

8. The computing device of claim 7 , wherein the instructions are further executable by the at least one processor to:

receive, from a plurality of client devices, a continuous stream of training instances that correspond to the candidate message displayed at the plurality of client devices, wherein each training instance indicates at least one of the impression callback or the engagement callback associated with the candidate message of the set of candidate messages; and

wherein the instructions that are executable by the at least one processor to train the machine learning model comprises the instructions being executable by the at least one processor to train the machine learning model based at least in part on the continuous stream of training instances that correspond to the candidate message.

9. The computing device of claim 6 , wherein the instructions that are executable by the at least one processor to determine the candidate score comprise instructions that are executable by the at least one processor to instructions that are executable by the at least one processor to:

determine, based at least in part on the probability that the user will select the candidate message and a bid price that an advertiser will pay if the user selects the candidate message, the candidate score for the candidate message.

10. The computing device of claim 6 , wherein the instructions that are executable by the at least one processor to send the candidate message comprise instructions that are executable by the at least one processor to:

send the candidate message in a set of candidate messages, wherein each respective candidate message in the set of candidate messages has a respective candidate score that satisfies the threshold score.

11. A non-transitory computer-readable storage medium for selecting a first candidate message from a set of candidate messages for display at a client device based on a probability, as determined by at least two learning models, that a user of the client device will select the first candidate message, the non-transitory computer-readable storage medium encoded with instructions that, when executed, cause at least one processor of a computing device to:

receive, from the client device of the user, a request for one or more candidate messages from the set of candidate messages to display at the client device together with a set of messages, wherein the set of messages is associated with the user in a social network messaging service;

calculate, using a machine learning model that is based on a point-wise learning model and a pair-wise learning model, a candidate score associated with the probability that the user will select the first candidate message from the set of candidate messages, wherein:

the point-wise learning model calculates a corresponding first candidate score for each candidate message of the set of candidate messages based on a first likelihood that the user of the client device will select that candidate message, and generates a ranking of the set of candidate messages based on the corresponding first candidate score for each candidate message;

the pair-wise learning model adjusts the first candidate score of at least one candidate message of the set of candidate messages based on a second likelihood that the user will select the at least one candidate message instead of another candidate message of the set of candidate messages, and reorders the ranking of the set of candidate messages based on the adjusted first candidate score of the at least one candidate message; and

the candidate score for the first candidate message is based on the reordered ranking of the set of candidate messages;

determine that the candidate score satisfies a threshold score;

send, for display at the client device, the first candidate message;

receive, from the client device, an impression callback time that indicates a time when the first candidate message was displayed at the client device;

receive, from the client device, an engagement callback time that indicates a time when the user selected the first candidate message at the client device; and

train the machine learning model based at least in part on a difference between the impression callback time and the engagement callback time.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the instructions that cause the at least one processor to train the machine learning model comprise instructions that, when executed, cause the at least one processor to:

train the machine learning model based on training instances for single candidate messages and training instances for pairs of candidate messages.

13. The non-transitory computer-readable storage medium of claim 12 , further comprising instructions that, when executed, cause the at least one processor to:

receive, from a plurality of client devices, a continuous stream of training instances that correspond to the candidate message displayed at the plurality of client devices, wherein each training instance indicates at least one of the impression callback or the engagement callback associated with the candidate message of the set of candidate messages; and

wherein the instructions that cause the at least one processor to train the machine learning model comprises instructions further cause the at least one processor to train the machine learning model based at least in part on the continuous stream of training instances that correspond to the candidate message.

14. The non-transitory computer-readable storage medium of claim 11 , wherein instructions that cause the at least one processor to determine the candidate score comprise instructions that, when executed, cause the at least one processor to:

determine, based at least in part on the probability that the user will select the candidate message and a bid price that an advertiser will pay if the user selects the candidate message, the candidate score for the candidate message.

15. The non-transitory computer-readable storage medium of claim 11 , further comprising instructions that, when executed, cause the at least one processor to:

send the candidate message in a set of candidate messages, wherein each respective candidate message in the set of candidate messages has a respective candidate score that satisfies the threshold score.