Model-based deep reinforcement learning for dynamic pricing in an online ride-hailing platform

1. A computer-implemented method for dynamic pricing, comprising:

training a reinforcement learning (RL) model in a simulation of a ride-hailing platform to predict a future state of the simulation based on a previous state of the simulation and a previous pricing action, wherein the training comprises minimizing a divergence between the predicted future state of the simulation and an observed futured state of the simulation, and the simulation is an offline version of the ride-hailing platform;

obtaining information comprising a set of pricing candidates and an initial status of the ride-hailing platform, wherein the initial status comprises a number of available drivers and a number of trip requests from passengers of the ride-hailing platform;

updating the set of pricing candidates based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform, wherein the updating comprises:

inputting the obtained information into the trained RL model;

obtaining a cumulative reward from the trained RL model for each pricing candidate in the set of pricing candidates based on the initial status of the ride-hailing platform;

obtaining a set of top pricing candidates comprising one or more pricing candidates with the highest cumulative rewards; and

updating the set of pricing candidates based on the set of top pricing candidates; and

generating a price for at least one current trip request on the ride-hailing platform based on the updated set of pricing candidates;

collecting an observed status of the ride-hailing platform after the generated price is deployed online;

retraining the RL model to reach a convergence between the prediction of the future status of the ride-hailing platform and the observed status of the ride-hailing platform.

2. The method of claim 1 , wherein the initial status of the ride-hailing platform further comprises a status of at least one origin-destination (OD) pair associated with an origin grid region and a destination grid region.

3. The method of claim 2 , wherein the status of the at least one OD pair comprises: a running estimate of a number of created orders, a number of accepted orders at a set of price increment, a number of finished orders at the set of price increment, and a number of canceled orders due to time running out.

4. The method of claim 1 , wherein the set of pricing candidates comprise a plurality of historical pricing sequences sampled from a set of historical pricing sequences.

5. The method of claim 1 , wherein the set of pricing candidates comprise a plurality of randomly generated pricing sequences.

6. The method of claim 1 , wherein obtaining the cumulative reward from the trained RL model comprises:

obtaining an initial state of the simulation of the ride-hailing platform based on the initial status of the ride-hailing platform;

obtaining a set of future states of the simulation of the ride-hailing platform based on the set of pricing candidates, the initial status of the ride-hailing platform, and the initial state of the simulation of the ride-hailing platform; and

generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform.

7. The method of claim 6 , wherein generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform comprises summing a state reward for each state of the set of future states of the simulation of the ride-hailing platform.

8. The method of claim 7 , wherein the trained RL model comprises:

a transition neural network configured for determining a prediction of each future state of the simulation of the ride-hailing platform based on a previous state of the simulation of the ride-hailing platform and a previous pricing action;

an observation neural network configured for determining a set of future observations of the simulation of the ride-hailing platform based on each future state of a simulation of the ride-hailing platform;

a state neural network configured for determining each future state of the simulation of the ride-hailing platform based on each future observation of the simulation of the ride-hailing platform; and

a reward neural network configured for determining each state reward based on each future state of the simulation of the ride-hailing platform.

9. The method of claim 8 , wherein the training further comprises maximizing a predicted accumulative reward.

10. The method of claim 7 , wherein the trained RL model comprises:

a recurrent neural network configured for determining a set of recurrent states of the simulation of the ride-hailing platform based on a set of pervious recurrent states of the simulation of the ride-hailing platform, a set of previous states of the simulation of the ride-hailing platform, and a set of previous pricing actions of the simulation of the ride-hailing platform;

a transition neural network configured for determining a prediction of each future state of the simulation of the ride-hailing platform based on each recurrent state of the simulation of the ride-hailing platform;

an observation neural network configured for determining a set of future observations of the simulation of the ride-hailing platform based on each recurrent state of the simulation of the ride-hailing platform and each future state of the simulation of the ride-hailing platform;

a state neural network configured for determining each future state of the simulation of the ride-hailing platform based on a set of previous observations the simulation of the ride-hailing platform and the set of previous pricing actions; and

a reward neural network configured for determining each state reward based on each recurrent state of the simulation of the ride-hailing platform and each future state of the simulation of the ride-hailing platform.

11. The method of claim 10 , wherein the training further comprises: maximizing a data log-likelihood of obtaining each future observation and each state reward.

12. The method of claim 1 , wherein generating the price for the at least one current trip request comprises generating a pricing multiplier for multiplying with a base price of the at least one current trip request.

13. A system for dynamic pricing, comprising one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations comprising:

training a reinforcement learning (RL) model in a simulation of a ride-hailing platform to predict a future state of the simulation based on a previous state of the simulation and a previous pricing action, wherein the training comprises minimizing a divergence between the predicted future state of the simulation and an observed futured state of the simulation;

obtaining information comprising a set of pricing candidates and an initial status of the ride-hailing platform, wherein the initial status comprises a number of available drivers and a number of trip requests from passengers of the ride-hailing platform;

updating the set of pricing candidates based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform, wherein the updating comprises:

inputting the obtained information into the trained RL model;

obtaining a cumulative reward from the trained RL model for each pricing candidate in the set of pricing candidates based on the initial status of the ride-hailing platform;

obtaining a set of top pricing candidates comprising one or more pricing candidates with the highest cumulative rewards; and

updating the set of pricing candidates based on the set of top pricing candidates; and

generating a price for at least one current trip request on the ride-hailing platform based on the updated set of pricing candidates;

collecting an observed status of the ride-hailing platform after the generated price is deployed;

retraining the RL model to reach a convergence between the prediction of the future status of the ride-hailing platform and the observed status of the ride-hailing platform.

14. The system of claim 13 , wherein obtaining the cumulative reward from the trained RL model comprises:

obtaining an initial state of the simulation of the ride-hailing platform based on the initial status of the ride-hailing platform;

obtaining a set of future states of the simulation of the ride-hailing platform based on the set of pricing candidates, the initial status of the ride-hailing platform, and the initial state of the simulation of the ride-hailing platform; and

generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform.

15. The system of claim 14 , wherein generating the cumulative reward based on the set of future states of the simulation of the ride-hailing platform comprises summing a state reward for each state of the set of future states of the simulation of the ride-hailing platform.

16. A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

training a reinforcement learning (RL) model in a simulation of a ride-hailing platform to predict a future state of the simulation based on a previous state of the simulation and a previous pricing action, wherein the training comprises minimizing a divergence between the predicted future state of the simulation and an observed futured state of the simulation;

obtaining information comprising a set of pricing candidates and an initial status of the ride-hailing platform, wherein the initial status comprises a number of available drivers and a number of trip requests from passengers of the ride-hailing platform;

updating the set of pricing candidates based on the initial status of the ride-hailing platform to minimize a cross-entropy between the set of pricing candidates and a target pricing policy that maximizes a total income of the ride-hailing platform, wherein the updating comprises:

inputting the obtained information into the trained RL model;

obtaining a cumulative reward from the trained RL model for each pricing candidate in the set of pricing candidates based on the initial status of the ride-hailing platform;

obtaining a set of top pricing candidates comprising one or more pricing candidates with the highest cumulative rewards; and

updating the set of pricing candidates based on the set of top pricing candidates; and

generating a price for at least one current trip request on the ride-hailing platform based on the updated set of pricing candidates;

collecting an observed status of the ride-hailing platform after the generated price is deployed;

retraining the RL model to reach a convergence between the prediction of the future status of the ride-hailing platform and the observed status of the ride-hailing platform.