Notification management and channel selection
Methods, systems, and computer programs are presented for predicting a response probability to a sent notification. One method includes an operation for training respective neural networks to obtain a first, second, and third models. The first model generates an embedding based on member information. The second and third model generate parameters for a distribution function. The first model is used to calculate a member embedding when accessing a notification for a member. Further, the method second model calculates a first parameter value, and the third model calculates a second parameter value based on the member embedding. Further, the method determines, a first probability that the member will visit the online service in response to the notification and a second probability that the member will visit without sending the notification. The method further includes determining to send the notification based on the first probability and the second probability.
1 . A computer-implemented method comprising:
training respective neural networks to obtain a first model, a second model, and a third model, the first model generating an embedding for a member of an online service based on member information, the second model generating a first parameter for a distribution function and the third model generating a second parameter for the distribution function, at least one of the second model and the third model trained on training data including notification timestamps, member visit timestamps, and a censor bit set, for each respective notification, to indicate whether the notification comprises an un-censored notification or a right-censored notification, thereby enabling the at least one of the second model and the third model to eliminate a bias in time-to-visit expectations for right-censored notifications;
accessing a notification for a first member;
calculating, by the first model, a first member embedding;
calculating, by the second model, a first parameter value based on the first member embedding;
calculating, by the third model, a second parameter value based on the first member embedding;
determining, based on the first parameter value and the second parameter value for the distribution function, a first probability that the first member will visit the online service in response to the notification and a second probability that the first member will visit the online service without sending the notification; and
determining to send the notification to the first member based on the first probability and the second probability.
2 . The method as recited in claim 1 , wherein the training is based on training data with values for features comprising member profile information, member activity, notifications and visits to the online service.
3 . The method as recited in claim 1 , wherein the distribution function is a Weibull distribution.
4 . The method as recited in claim 3 , wherein the first parameter is a shape of the Weibull distribution and the second parameter is a scale of the Weibull distribution.
5 . The method as recited in claim 4 , wherein a loss function of the second model is based on a log of the shape of the Weibull distribution.
6 . The method as recited in claim 4 , wherein a loss function of the third model is based on a log of the scale of the Weibull distribution.
7 . The method as recited in claim 1 , wherein determining to send the notification further comprises:
calculating a difference between the first probability and the second probability; and
sending the notification when the difference is greater than a predetermined threshold value.
8 . The method as recited in claim 1 , further comprising:
requeuing the notification for later processing when the determination is not to send the notification.
9 . The method as recited in claim 1 , wherein the first and second probability are calculated for a plurality of channels comprising badge notifications, push notifications, and emails.
10 . The method as recited in claim 1 , wherein the embedding generated by the first model is an input for the second model and the third model.
11 . A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
training respective neural networks to obtain a first model, a second model, and a third model, the first model generating an embedding for a member of an online service based on member information, the second model generating a first parameter for a distribution function and the third model generating a second parameter for the distribution function, at least one of the second model and the third model trained on training data including notification timestamps, member visit timestamps, and a censor bit set, for each respective notification, to indicate whether the notification comprises an un-censored notification or a right-censored notification, thereby enabling the at least one of the second model and the third model to eliminate a bias in time-to-visit expectations for right-censored notifications;
accessing a notification for a first member;
calculating, by the first model, a first member embedding;
calculating, by the second model, a first parameter value based on the first member embedding;
calculating, by the third model, a second parameter value based on the first member embedding;
determining, based on the first parameter value and the second parameter value for the distribution function, a first probability that the first member will visit the online service in response to the notification and a second probability that the first member will visit the online service without sending the notification; and
determining to send the notification based on the first probability and the second probability.
12 . The system as recited in claim 11 , wherein the training is based on training data with values for features comprising member profile information, member activity, notifications and visits to the online service.
13 . The system as recited in claim 11 , wherein the distribution function is a Weibull distribution.
14 . The system as recited in claim 13 , wherein the first parameter is a shape of the Weibull distribution and the second parameter is a scale of the Weibull distribution, wherein a loss function of the second model is based on a log of the shape of the Weibull distribution, wherein a loss function of the third model is based on a log of the scale of the Weibull distribution.
15 . The system as recited in claim 11 , wherein determining to send the notification further comprises:
calculating a difference between the first probability and the second probability; and
sending the notification when the difference is greater than a predetermined threshold value.
16 . A tangible machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
training respective neural networks to obtain a first model, a second model, and a third model, the first model generating an embedding for a member of an online service based on member information, the second model generating a first parameter for a distribution function and the third model generating a second parameter for the distribution function, at least one of the second model and the third model trained on training data including notification timestamps, member visit timestamps, and a censor bit set, for each respective notification, to indicate whether the notification comprises an un-censored notification or a right-censored notification, thereby enabling the at least one of the second model and the third model to eliminate a bias in time-to-visit expectations for right-censored notifications;
accessing a notification for a first member;
calculating, by the first model, a first member embedding;
calculating, by the second model, a first parameter value based on the first member embedding;
calculating, by the third model, a second parameter value based on the first member embedding;
determining, based on the first parameter value and the second parameter value for the distribution function, a first probability that the first member will visit the online service in response to the notification and a second probability that the first member will visit the online service without sending the notification; and
determining to send the notification based on the first probability and the second probability.
17 . The tangible machine-readable storage medium as recited in claim 16 , wherein the training is based on training data with values for features comprising member profile information, member activity, notifications and visits to the online service.
18 . The tangible machine-readable storage medium as recited in claim 16 , wherein the distribution function is a Weibull distribution.
19 . The tangible machine-readable storage medium as recited in claim 18 , wherein the first parameter is a shape of the Weibull distribution and the second parameter is a scale of the Weibull distribution, wherein a loss function of the second model is based on a log of the shape of the Weibull distribution, wherein a loss function of the third model is based on a log of the scale of the Weibull distribution.
20 . The tangible machine-readable storage medium as recited in claim 16 , wherein determining to send the notification further comprises:
calculating a difference between the first probability and the second probability; and
sending the notification when the difference is greater than a predetermined threshold value.