Reinforcement machine learning framework for dynamic demand forecasting
Systems and methods for implementing a reinforcement machine learning framework for dynamic demand forecasting. A method includes generating estimated booking data for an initial time with a demand model trained using a training set of historical booking data. A variance is detected between the estimated booking data and transient booking data observed at the initial time that exceeds a defined threshold. In response to detecting the variance, a reinforcement learning service is activated. An updated training set including enhanced booking data observed at a subsequent time is created after activating the reinforcement learning service. A parameter of the demand model is updated by training the demand model using the updated training set.
1 . A method comprising:
generating estimated booking data for an initial time with a demand model trained using a training set of historical booking data;
obtaining transient booking data observed at the initial time, the transient booking data comprising booking records for travel services scheduled to depart on future dates;
detecting, by an auditing service, a variance between the estimated booking data and the transient booking data observed at the initial time that exceeds a defined threshold;
in response to detecting the variance, activating an exploration mode of a reinforcement learning service in response to detecting the variance, wherein the reinforcement learning service is configured to dynamically adjust exploration and exploitation in response to detected demand shocks, wherein the exploration mode comprises modifying offer frequencies or prices of multiple fare classes to increase diversity of observed price-demand pairs, wherein the reinforcement learning service intercepts one or more inventory control attributes selected by an optimization service and processes the attributes based on an exploration-exploitation strategy to increase price disparity within an updated training set by increasing a diversity of observations of price-demand pairs;
creating an updated training set comprising enhanced booking data, wherein the enhanced booking data includes booking records updated to reflect additional observations obtained during the exploration mode thereby increasing the diversity of observations of price-demand pairs in the historical booking data, wherein creating the updated training set comprises using an offer threshold process configured to effectuate the exploration mode by modifying offer frequencies of multiple fare classes based on a respective number of offers for each fare class identified within the historical booking data; and
updating a parameter of the demand model by training the demand model using the updated training set.
2 . The method of claim 1 , wherein generating the estimated booking data comprises:
computing a probability that a travel service or a flight will receive a given number of bookings by a given day-to-departure based on a demand forecast obtained using the demand model.
3 . The method of claim 1 , wherein the reinforcement learning service is configured to increase price disparity within an updated training set at a subsequent time based on increasing a diversity of observations of price-demand pairs.
4 . The method of claim 1 , wherein the reinforcement learning service is configured to compute an optimal probability of transitioning to the exploration mode to maximize a net value of a gain of information minus a cost of exploration, wherein the gain of information comprises an expected improvement in demand model accuracy as a result of increased price diversity in an updated training set based on the exploration mode, and the cost of exploration comprises a difference in expected revenue between offering an exploratory fare class selected by the exploration mode and offering the fare class that would have been selected by an exploitation mode.
5 . The method of claim 1 , wherein modifying the offer frequencies of the multiple fare classes comprises:
reducing an offer frequency of a given fare class with a highest number of offers among the multiple fare classes.
6 . The method of claim 1 , wherein modifying the offer frequencies of the multiple fare classes comprises:
increasing an offer frequency of a given fare class with less than a pre-defined minimum number of offers.
7 . The method of claim 1 , wherein the exploration-exploitation strategy comprises an operating mode of transitioning between the exploration mode and an exploitation mode.
8 . The method of claim 1 , further comprising:
generating a confidence cone by aggregating a set of probabilities computed for multiple flights with each probability estimating a likelihood that a given flight among the multiple flights will receive a given number of bookings by a given day-to-departure based on a demand forecast obtained using the demand model.
9 . The method of claim 1 , further comprising:
updating the training set by:
computing an entropy of the training set to determine an uncertainty of the demand model;
identifying fare classes to explore to minimize the entropy in the updated training set;
estimating a cost of exploring at each fare class, comprised of a difference in expected revenue between offering an exploratory fare class and offering the fare class that would have been selected by an exploitation mode; and
selecting a fare class to explore that maximizes an expected improvement in demand model accuracy as a result of increased price diversity in the updated training set based on a maximum exploration cost threshold.
10 . The method of claim 1 , wherein the parameter comprises a price elasticity parameter, a product attractiveness parameter, a demand volume parameter, or a temporal adjustment parameter.
11 . A system comprising:
one or more processors;
at least one memory device coupled with the one or more processors; and
a data communications interface operably associated with the one or more processors, wherein the at least one memory device contains a plurality of program instructions that, when executed by the one or more processors, cause the system to:
generate estimated booking data for an initial time with a demand model trained using a training set of historical booking data;
obtain transient booking data observed at the initial time, the transient booking data comprising booking records for travel services scheduled to depart on future dates;
detect, by an auditing service, a variance between the estimated booking data and the transient booking data observed at the initial time that exceeds a defined threshold;
in response to detect the variance, activate an exploration mode of a reinforcement learning service in response to detecting the variance, wherein the reinforcement learning service is configured to dynamically adjust exploration and exploitation in response to detected demand shocks, wherein the exploration mode comprises modifying offer frequencies or prices of multiple fare classes to increase diversity of observed price-demand pairs, wherein the reinforcement learning service intercepts one or more inventory control attributes selected by an optimization service and processes the attributes based on an exploration-exploitation strategy to increase price disparity within an updated training set by increasing a diversity of observations of price-demand pairs;
create an updated training set comprising enhanced booking data, wherein the enhanced booking data includes booking records updated to reflect additional observations obtained during the exploration mode thereby increasing the diversity of observations of price-demand pairs in the historical booking data, wherein creating the updated training set comprises using an offer threshold process configured to effectuate the exploration mode by modifying offer frequencies of multiple fare classes based on a respective number of offers for each fare class identified within the historical booking data; and
update a parameter of the demand model by training the demand model using the updated training set.
12 . The system of claim 11 , wherein the plurality of program instructions cause the system to generate the estimated booking data by causing the system to compute a probability that a travel service or flight will receive a given number of bookings by a given day-to-departure based on a demand forecast obtained using the demand model.
13 . The system of claim 11 , wherein the reinforcement learning service is configured to increase price disparity within an updated training set based on increasing a diversity of observations of price-demand pairs.
14 . The system of claim 11 , wherein the reinforcement learning service is configured to compute an optimal probability of transitioning to the exploration mode to maximize a net value of a gain of information minus a cost of exploration.
15 . The system of claim 11 , wherein the program instructions, when executed by the one or more processors, further cause the system to update the training set by causing the system to:
compute an entropy of the training set to determine an uncertainty of the demand model;
identify fare classes to explore to minimize the entropy in the updated training set;
estimate a cost of exploring at each fare class, comprised of a difference in expected revenue between offering an exploratory fare class and offering the fare class that would have been selected by an exploitation mode; and
select a fare class to explore that maximizes an expected improvement in demand model accuracy as a result of increased price diversity in the updated training set based on a maximum exploration cost threshold.
16 . A computer program product comprising:
a non-transitory computer-readable storage medium; and
program code stored on the non-transitory computer-readable storage medium that, when executed by one or more processors, causes the one or more processors to:
generate estimated booking data for an initial time with a demand model trained using a training set of historical booking data;
obtain transient booking data observed at the initial time, the transient booking data comprising booking records for travel services scheduled to depart on future dates;
detect, by an auditing service, a variance between the estimated booking data and the transient booking data observed at the initial time that exceeds a defined threshold;
in response to detect the variance, activate an exploration mode of a reinforcement learning service in response to detecting the variance, wherein the reinforcement learning service is configured to dynamically adjust exploration and exploitation in response to detected demand shocks, wherein the exploration mode comprises modifying offer frequencies or prices of multiple fare classes to increase diversity of observed price-demand pairs, wherein the reinforcement learning service intercepts one or more inventory control attributes selected by an optimization service and processes the attributes based on an exploration-exploitation strategy to increase price disparity within an updated training set by increasing a diversity of observations of price-demand pairs;
create an updated training set comprising enhanced booking data, wherein the enhanced booking data includes booking records updated to reflect additional observations obtained during the exploration mode thereby increasing the diversity of observations of price-demand pairs in the historical booking data, wherein creating the updated training set comprises using an offer threshold process configured to effectuate the exploration mode by modifying offer frequencies of multiple fare classes based on a respective number of offers for each fare class identified within the historical booking data; and
update a parameter of the demand model by training the demand model using the updated training set.
17 . The method of claim 1 , wherein the exploration mode is distinct from an exploitation mode that maximizes immediate revenue based on current demand estimates.
18 . The method of claim 1 , wherein creating the updated training set comprises determining a current uncertainty of the demand model, and driving targeted exploration by identifying fare classes whose additional observations that reduce that uncertainty and lower entropy in the updated training set.
19 . The method of claim 1 , wherein, in response to detected demand shocks, the auditing service initiates the reinforcement learning service in exploration mode and provides one or more contextual inputs that guide exploration-exploitation transitions.
20 . The method of claim 19 , wherein the one or more contextual inputs comprise at least one of magnitude, timing, and affected fare classes.