Machine learned model for proactively selecting a remedial action using before receiving a notification of a problem with an order
An online concierge system fulfills orders placed by users. When a user notifies the online concierge system of a problem with order fulfillment, the online concierge system performs one or more remedial actions (e.g., a credit, a discount, a free delivery). To provide a proactive remedial action before receiving a notification of a problem with order fulfillment from a user, the online concierge system trains a proactive remediation model that predicts, for an order having an event during fulfillment, a likelihood of loss of interaction by the user with the online concierge system (i.e., “churn” of the user) without performing a proactive remedial action. When fulfilling an order, in response to determining an event during fulfillment, the online concierge system applies the proactive remediation model to determine the likelihood of churn of the user if no proactive appeasement is performed, for determining whether to perform a proactive remedial action.
1 . A method, performed at a computer system comprising a processor and a non-transitory computer readable medium, comprising:
receiving, at the computer system, an order from a user, the order including one or more items to obtain from a retailer;
receiving, at the computer system, information describing fulfillment of the order from a picker fulfilling the order;
identifying, from the information describing fulfillment of the order, a problem with fulfillment of the order;
generating a probability of churn of the user occurring without the computer system performing a remedial action before receiving a notification of the problem with fulfillment of the order from the user by applying a proactive remediation model to characteristics of the user and to attributes of the order, wherein the proactive remediation model is a classification neural network that is trained by:
obtaining a training dataset including a plurality of training examples, each training example including characteristics of a training user and attributes of a prior order fulfilled for the training user, each training example having a label indicating whether churn occurred for the training user in response to fulfillment of the prior order, wherein obtaining the training dataset comprises generating an order issue attribute of the attributes of the prior order for each of the plurality of training examples, wherein generating the order issue attribute for a training example comprises:
receiving image data captured by a camera of a client device associated with a picker who serviced the prior order of the training example, wherein the image data depicts a set of items picked by the picker for the prior order;
applying a computer-vision machine-learning model to the image data to generate a prediction for the set of items picked by the picker for the prior order;
comparing the prediction for the set of items to a set of items ordered by the training user to identify at least one item in the set of items ordered by the user that is not in the prediction of the set of items; and
responsive to identifying the at least one item, generating the order issue attribute for the training example to indicate an issue with the prior order;
wherein obtaining the training dataset comprises generating the label indicating whether churn occurred for a training example by:
measuring a first rate of user interaction with a user interface of a client application associated with an online system executing on a client device of the user, wherein the first rate of user interaction is measured before placement of the prior order associated with the training example;
measuring a second rate of user interaction with the user interface of the client application associated with the online system executing on the client device of the user, wherein the second rate of user interaction is measured after placement of the prior order associated with the training example;
comparing the first rate of user interaction with the second rate of user interaction to determine a decrease in the rate of user interaction from the first rate to the second rate; and
responsive to determining the decrease, generating the label to indicate a churn for the user;
applying the proactive remediation model to each training example of the training dataset to generate a predicted probability of churn of the training user occurring without providing a remedial action before the computer system receives a notification of a problem with fulfilment of the prior order from the training user;
scoring the proactive remediation model using a loss function and the label of the score training example; and
updating one or more parameters of the proactive remediation model by backpropagation based on the scoring until one or more criteria are satisfied;
generating an expected value for one or more candidate remedial actions based on the probability of churn of the user occurring, the expected value for the candidate remedial action based on a cost of the candidate remedial action to the computing system and the probability of churn from the proactive remediation model;
selecting a candidate remedial action based on the expected values; and
transmitting the selected candidate remedial action to a user client device of the user before the computer system receives the notification of the problem with order fulfillment from the user.
2 . The method of claim 1 , wherein transmitting the selected candidate remedial action to the user client device of the user comprises:
transmitting the selected candidate remedial action to the user client device of the user at a time when the picker delivered the one or more items to a location included in the order.
3 . The method of claim 1 , wherein transmitting the selected candidate remedial action to the user client device of the user comprises:
transmitting the selected candidate remedial action to the user client device of the user at a time before the picker delivered the one or more items to a location included in the order.
4 . The method of claim 1 , wherein selecting the candidate remedial action comprises selecting one or more of: providing a credit to the user, providing a discount for the order to the user, providing a discount for a subsequent order to the user, or providing free delivery of the items in the order to the user.
5 . The method of claim 1 , wherein applying the proactive remediation model to characteristics of the user comprises applying the proactive remediation model to is one or more of: a churn rate of the user determined from orders previously fulfilled for the user, a number of orders previously received from the user, or a cumulative number of orders received from the user, or an indication whether a count of orders received from the user is less than a threshold number.
6 . The method of claim 1 , wherein applying the proactive remediation model to attributes of the order comprises applying the proactive remediation model to one or more of: a rate at which an item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order, a type of the item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order, a type of retailer included in the order, an order type of the order, or a tip amount specified by the user for the picker fulfilling the order.
7 . The method of claim 6 , wherein the type of item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order comprises an indication of whether the item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order is a foundational item for the order.
8 . The method of claim 1 , wherein applying the proactive remediation model to attributes of the order comprises applying the proactive remediation model to one or more of: a description of describes weather when the order was fulfilled, or a holiday indication whether a holiday occurs within a threshold amount of time from a time for fulfillment indicated in the order.
9 . The method of claim 1 , wherein generating the expected value for one or more candidate remedial actions based on the probability of churn of the user occurring comprises:
generating the expected value for one or more candidate remedial actions based on the probability of churn of the user occurring in response to the probability of churn of the user occurring equaling or exceeding a threshold value.
10 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
receiving, at the computer system, an order from a user, the order including one or more items to obtain from a retailer;
receiving, at the computer system, information describing fulfillment of the order from a picker fulfilling the order;
identifying, from the information describing fulfillment of the order, a problem with fulfillment of the order;
generating a probability of churn of the user occurring without the computer system performing a remedial action before receiving a notification of the problem with fulfillment of the order from the user by applying a proactive remediation model to characteristics of the user and to attributes of the order, wherein the proactive remediation model is a classification neural network that is trained by:
obtaining a training dataset including a plurality of training examples, each training example including characteristics of a training user and attributes of a prior order fulfilled for the training user, each training example having a label indicating whether churn occurred for the training user in response to fulfillment of the prior order, wherein obtaining the training dataset comprises generating an order issue attribute of the attributes of the prior order for each of the plurality of training examples, wherein generating the order issue attribute for a training example comprises:
receiving image data captured by a camera of a client device associated with a picker who serviced the prior order of the training example, wherein the image data depicts a set of items picked by the picker for the prior order;
applying a computer-vision machine-learning model to the image data to generate a prediction for the set of items picked by the picker for the prior order;
comparing the prediction for the set of items to a set of items ordered by the user to identify at least one item in the set of items ordered by the user that is not in the prediction of the set of items; and
responsive to identifying the at least one item, generating the order issue attribute for the training example to indicate an issue with the prior order;
wherein obtaining the training dataset comprises generating the label indicating whether churn occurred for a training example by:
measuring a first rate of user interaction with a user interface of a client application associated with the online system executing on a client device of the user, wherein the first rate of user interaction is measured before placement of the prior order associated with the training example;
measuring a second rate of user interaction with the user interface of the client application associated with the online system executing on the client device of the user, wherein the second rate of user interaction is measured after placement of the prior order associated with the training example;
comparing the first rate of user interaction with the second rate of user interaction to determine a decrease in the rate of user interaction from the first rate to the second rate; and
responsive to determining the decrease, generating the label to indicate a churn for the user;
applying the proactive remediation model to each training example of the training dataset to generate a predicted probability of churn of the training user occurring without providing a remedial action before the computer system receives a notification of a problem with fulfilment of the prior order from the training user;
scoring the proactive remediation model using a loss function and the label of the score training example; and
updating one or more parameters of the proactive remediation model by backpropagation based on the scoring until one or more criteria are satisfied;
generating an expected value for one or more candidate remedial actions based on the probability of churn of the user occurring, the expected value for the candidate remedial action based on a cost of the candidate remedial action to the computing system and the probability of churn from the proactive remediation model;
selecting a candidate remedial action based on the expected values; and
transmitting the selected candidate remedial action to a user client device of the user before the computer system receives the notification of the problem with order fulfillment from the user.
11 . The computer program product of claim 10 , wherein transmitting the selected candidate remedial action to the user client device of the user comprises:
transmitting the selected candidate remedial action to the user client device of the user at a time when the picker delivered the one or more items to a location included in the order.
12 . The computer program product of claim 10 , wherein transmitting the selected candidate remedial action to the user client device of the user comprises:
transmitting the selected candidate remedial action to the user client device of the user at a time before the picker delivered the one or more items to a location included in the order.
13 . The computer program product of claim 10 , wherein selecting the candidate remedial action comprises selecting one or more of: providing a credit to the user, providing a discount for the order to the user, providing a discount for a subsequent order to the user, or providing free delivery of the items in the order to the user.
14 . The computer program product of claim 10 , wherein applying the proactive remediation model to characteristics of the user comprises applying the proactive remediation model to is one or more of: a churn rate of the user determined from orders previously fulfilled for the user, a number of orders previously received from the user, or a cumulative number of orders received from the user, or an indication whether a count of orders received from the user is less than a threshold number.
15 . The computer program product of claim 10 , wherein applying the proactive remediation model to attributes of the order comprises applying the proactive remediation model to one or more of: a rate at which an item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order, a type of the item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order, a type of retailer included in the order, an order type of the order, or a tip amount specified by the user for the picker fulfilling the order.
16 . The computer program product of claim 15 , wherein the type of item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order comprises an indication of whether the item in the order that the computer system determined was unable to be obtained from the retailer included in the order when identifying the problem with fulfillment of the order is a foundational item for the order.
17 . The computer program product of claim 10 , wherein applying the proactive remediation model to attributes of the order comprises applying the proactive remediation model to one or more of: a description of describes weather when the order was fulfilled, or a holiday indication whether a holiday occurs within a threshold amount of time from a time for fulfillment indicated in the order.
18 . The computer program product of claim 10 , wherein generating the expected value for one or more candidate remedial actions based on the probability of churn of the user occurring comprises:
generating the expected value for one or more candidate remedial actions based on the probability of churn of the user occurring in response to the probability of churn of the user occurring equaling or exceeding a threshold value.
19 . A system comprising:
a processor; and
a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the processor, cause the processor to perform steps comprising:
receiving, at the computer system, an order from a user, the order including one or more items to obtain from a retailer;
receiving, at the computer system, information describing fulfillment of the order from a picker fulfilling the order;
identifying, from the information describing fulfillment of the order, a problem with fulfillment of the order;
generating a probability of churn of the user occurring without the computer system performing a remedial action before receiving a notification of the problem with fulfillment of the order from the user by applying a proactive remediation model to characteristics of the user and to attributes of the order, wherein the proactive remediation model is a classification neural network that is trained by:
obtaining a training dataset including a plurality of training examples, each training example including characteristics of a training user and attributes of a prior order fulfilled for the training user, each training example having a label indicating whether churn occurred for the training user in response to fulfillment of the prior order, wherein obtaining the training dataset comprises generating an order issue attribute of the attributes of the prior order for each of the plurality of training examples, wherein generating the order issue attribute for a training example comprises:
receiving image data captured by a camera of a client device associated with a picker who serviced the prior order of the training example, wherein the image data depicts a set of items picked by the picker for the prior order;
applying a computer-vision machine-learning model to the image data to generate a prediction for the set of items picked by the picker for the prior order;
comparing the prediction for the set of items to a set of items ordered by the user to identify at least one item in the set of items ordered by the user that is not in the prediction of the set of items; and
responsive to identifying the at least one item, generating the order issue attribute for the training example to indicate an issue with the prior order;
wherein obtaining the training dataset comprises generating the label indicating whether churn occurred for a training example by:
measuring a first rate of user interaction with a user interface of a client application associated with the online system executing on a client device of the user, wherein the first rate of user interaction is measured before placement of the prior order associated with the training example;
measuring a second rate of user interaction with the user interface of the client application associated with the online system executing on the client device of the user, wherein the second rate of user interaction is measured after placement of the prior order associated with the training example;
comparing the first rate of user interaction with the second rate of user interaction to determine a decrease in the rate of user interaction from the first rate to the second rate; and
responsive to determining the decrease, generating the label to indicate a churn for the user;
applying the proactive remediation model to each training example of the training dataset to generate a predicted probability of churn of the training user occurring without providing a remedial action before the computer system receives a notification of a problem with fulfilment of the prior order from the training user;
scoring the proactive remediation model using a loss function and the label of the score training example; and
updating one or more parameters of the proactive remediation model by backpropagation based on the scoring until one or more criteria are satisfied;
generating an expected value for one or more candidate remedial actions based on the probability of churn of the user occurring, the expected value for the candidate remedial action based on a cost of the candidate remedial action to the computing system and the probability of churn from the proactive remediation model;
selecting a candidate remedial action based on the expected values; and
transmitting the selected candidate remedial action to a user client device of the user before the computer system receives the notification of the problem with order fulfillment from the user.
20 . The system of claim 19 , wherein transmitting the selected candidate remedial action to the user client device of the user comprises:
transmitting the selected candidate remedial action to the user client device of the user at a time when the picker delivered the one or more items to a location included in the order.