Optimizing send time for electronic communications
An improved electronic communication system schedules transmission of electronic communications based on a predicted open time and click time. The open and click times are predicted from a machine learning model that is trained to optimize for both tasks. Additionally, when training the machine learning model, the loss used for adjusting the system to achieve a desired accuracy may be a biased loss determined from a function that penalizes overpredicting the open time. As such, the loss value may be determined by different set of rules depending on whether the predicted time is greater than the actual time or not.
1 . A computer-implemented method comprising:
receiving an indication of an electronic communication to be sent to a recipient, the electronic communication including a link selectable by the recipient;
based on recipient data for a plurality of recipients including the recipient, determining a predicted open time and a predicted click time of the link selectable by the recipient included in the electronic communication, wherein the predicted open time and the predicted click time are determined from an output of a machine learning model, the machine learning model including a neural network with a first branch of the neural network that outputs predicted open times and a second branch of the neural network that outputs predicted click times, wherein the neural network is trained using a joint loss value based on a click time loss associated with the predicted click times and an open time loss associated with the predicted open times, where the open time loss is weighted more heavily than the click time loss, wherein the machine learning model is trained based on a first biased loss value generated using a first loss function that penalizes predicting open times later than ground truth open times and a second biased loss value generated using a second loss function that penalizes predicting click times later than ground truth click times;
based on the predicted open time and the predicted click time, determining a send time for the electronic communication;
based on the send time, generating a schedule for sending the electronic communication to the recipient; and
causing the electronic communication to be sent to the recipient based on the schedule.
2 . The computer-implemented method of claim 1 , wherein determining the send time for the electronic communication comprises applying one or more predefined rules to the predicted open time and the predicted click time.
3 . The computer-implemented method of claim 1 , wherein determining the send time comprises determining an earlier of the predicted open time and the predicted click time and setting the send time to a pre-determined period of time prior to the earlier of the predicted open time and the predicted click time.
4 . The computer-implemented method of claim 1 , wherein the electronic communication is an e-mail.
5 . The computer-implemented method of claim 1 , wherein determining the predicted open time and the predicted click time comprises determining a plurality of predicted open times and a plurality of predicted click times based on recipient data for a plurality of recipients, each predicted open time and each predicted click time corresponding to a particular recipient; wherein determining the send time comprises determining a plurality of send times.
6 . The computer-implemented method of claim 1 , wherein the recipient data comprises one or more of data identifying characteristics of the recipient, historical electronic communication data of the recipient, data identifying characteristics of an organization to which the recipient belongs, and historical electronic communication data of the organization.
7 . The computer-implemented method of claim 1 , wherein causing the electronic communication to be sent to the recipient based on the schedule is performed without a user action.
8 . The computer-implemented method of claim 1 , wherein the click time loss value is determined based on the predicted click time and a ground truth click time indicated in ground truth data, where a penalty value is applied to a difference between the predicted click time and the ground truth click time.
9 . The computer-implemented method of claim 1 , wherein the open time loss value is determined based on the predicted open time and a ground truth open time indicated in ground truth data, where a penalty value is applied to a difference between the predicted open time and the ground truth open time.
10 . The computer-implemented method of claim 1 , wherein the joint loss value is determined by at least combining the click time loss and the open time loss with a weight value.
11 . One or more non-transitory computer storage media storing computer-readable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
receiving training recipient data associated with a recipient of an electronic communication, the electronic communication including a link that is selectable by the recipient;
utilizing a machine learning model to determine a predicted open time and a predicted click time associated with the link within the electronic communication based on at least the training recipient data, wherein the machine learning model includes a neural network with a first branch of the neural network that outputs the predicted open time and a second branch of the neural network that outputs the predicted click time;
determining a joint loss value based on a click time loss associated with the predicted click time and an open time loss associated with the predicted open time, where the open time loss is determined based on a first biased loss function taking as a first input a ground truth open time corresponding to a first interval of time the recipient opened the electronic communication indicated in the training recipient data, the first biased loss function penalizing predicting open times later than ground truth open times and a second biased loss function penalizing predicting click times later than ground truth click times, and the click time loss is determined based on the second biased loss function taking as a second input a ground truth click time corresponding to a second interval of time the recipient selected the link in the electronic communication, where the ground truth open time and the ground truth click time are indicated in the training recipient data;
adjusting the machine learning model based on the joint loss value;
determining a send time for a second electronic communication based on a second predicted open time and a second predicted click time determined by the machine learning model;
generating a schedule for sending the second electronic communication based on the send time; and
causing the second electronic communication to be sent based on the schedule.
12 . The computer storage media of claim 11 , wherein the joint loss value is a weighted sum of an open time loss value and a click time loss value, the open time loss value being based on a comparison of the predicted open time and the ground truth open time and the click time loss value being based on a comparison of the predicted click time and the ground truth click time.
13 . The computer storage media of claim 12 , wherein determining the open time loss value comprises determining whether the predicted open time is greater than the ground truth open time, and based on a determination that the predicted open time is greater than the ground truth open time, applying a penalty value.
14 . The computer storage media of claim 11 , wherein the joint loss value is determined by at least combining the click time loss and the open time loss with a weight value.
15 . A computing system comprising:
a processor;
memory storing instructions;
a data store comprising training recipient data and ground truth data; and
a means for training a machine learning model to predict when an electronic communication will be opened by an intended recipient, wherein the machine learning model is trained to bias towards predicting earlier open times, by at least:
utilizing a first branch of the machine learning model, determining a predicted open time based on training recipient data corresponding to the intended recipient;
utilizing a second branch of the machine learning model, determining a predicted click time based on training recipient data corresponding to the intended recipient;
based on the predicted open time and a ground truth open time, determining a biased loss value using a loss function that penalizes predicting open times later than ground truth open times;
based on the predicted click time and a ground truth click time, determining a second biased loss value using a second loss function that penalizes predicting click times later than ground truth click times;
adjusting the machine learning model based on the biased loss value and the second biased loss value;
determining a send time for a second electronic communication based on a second predicted open time and a second predicted click time determined by the machine learning model;
generating a schedule for sending the second electronic communication based on the send time; and
causing the second electronic communication to be sent based on the schedule.
16 . The computing system of claim 15 , wherein the machine learning model comprises a neural network system.
17 . The computing system of claim 15 , wherein determining the biased loss value comprises determining that the predicted open time is later than the ground truth open time and applying a penalty value to a difference between the predicted open time and the ground truth open time.
18 . The computing system of claim 15 , wherein the predicted click time indicates when content within the electronic communication will be selected.
19 . The computing system of claim 18 , wherein the means for training the machine learning model comprises utilizing training recipient data to determine contemporaneously the predicted open time and the predicted click time, and determining a joint loss value by combining the first biased loss value and the second biased loss value.
20 . The computing system of claim 19 , wherein the click time loss value is determined based on the predicted click time and a ground truth click time indicated in the ground truth data.