Automated policy function adjustment using reinforcement learning algorithm

1 . A method for creating an adaptive autonomous system, the method comprising:

at an online concierge system comprising a processor and a computer-readable medium:

receiving, by the online concierge system from a computer system associated with a content provider, a content presentation campaign that includes one or more objectives set by the content provider, at least one of the objectives defining a desired outcome;

defining a set of one or more policy functions that automatically control the content presentation campaign, each policy function comprising one or more parameters and a plurality of states, each state corresponding to a previously used parameter configuration and an associated realized outcome of the content presentation campaign;

monitoring a realized outcome of the content presentation campaign that is controlled by the set of policy functions;

storing, by the online concierge system, data indicating the realized outcomes and the corresponding states of the policy functions associated with the campaign;

applying a reinforcement learning algorithm that autonomously adjusts one or more parameters of the policy functions using a state-aware counterfactual estimation process that references previously stored states and corresponding outcomes to reduce a difference between the realized outcomes and the desired outcome set by the content provider;

generating an adjusted set of policy functions by the reinforcement learning algorithm; and

using the adjusted set of policy functions to control subsequent content selection operations of the online concierge system, thereby enabling autonomous system adaptation to changing campaign performance.

2 . The method of claim 1 , wherein using the adjusted set of policy functions comprises inputting estimates of a content slot to a first policy function of the policy functions to generate a bid value.

3 . The method of claim 2 , wherein inputting estimates of a content slot to the first policy function to generate the bid value comprises:

generating a plurality of features related to the content slot;

inputting the plurality of features to a machine learning model to predict the estimate; and

using the estimates generated by the machine learning model as inputs to the first policy function.

4 . The method of claim 1 , wherein the set of policy functions comprises a plurality of policy functions, each policy function being defined based on an objective provided by the content provider.

5 . The method of claim 1 , wherein at least one of the policy functions comprises a plurality of states that record past actions and outcomes associated with the policy function.

6 . The method of claim 1 , wherein the reinforcement learning algorithm updates the set of policy functions through a counterfactual policy estimation.

7 . The method of claim 1 , wherein the reinforcement learning algorithm is heuristic based, and wherein applying the heuristic based reinforcement learning algorithm comprises:

defining a rule in adjusting a parameter in a policy function;

examining the policy function at a previous state that has a known realized outcome; and

adjusting the parameter based on the rule and the known realized outcome of the previous state.

8 . The method of claim 1 , wherein the reinforcement learning algorithm is a machine learning based.

9 . The method of claim 1 , wherein one of the content items in the content presentation campaign is a sponsored item offered on one or more interfaces hosted by the online concierge system.

10 . A non-transitory computer readable medium configured to store code comprising instructions, the instructions, when executed by one or more processors, cause the one or more processors to:

receive, by an online concierge system from a computer system associated with a content provider, a content presentation campaign that includes one or more objectives set by the content provider, at least one of the objectives defining a desired outcome;

define a set of one or more policy functions that automatically control the content presentation campaign, each policy function comprising one or more parameters and a plurality of states, each state corresponding to a previously used parameter configuration and an associated realized outcome of the content presentation campaign;

monitor a realized outcome of the content presentation campaign that is controlled by the set of policy functions;

store, by the online concierge system, data indicating the realized outcomes and the corresponding states of the policy functions associated with the campaign;

apply a reinforcement learning algorithm that autonomously adjusts one or more parameters of the policy functions using a state-aware counterfactual estimation process that references previously stored states and corresponding outcomes to reduce a difference between the realized outcomes and the desired outcome set by the content provider;

generate an adjusted set of policy functions by the reinforcement learning algorithm; and

use the adjusted set of policy functions to control subsequent content selection operations of the online concierge system, thereby enabling autonomous system adaptation to changing campaign performance.

11 . The non-transitory computer readable medium of claim 10 , wherein using the adjusted set of policy functions comprises inputting estimates of a content slot to a first policy function of the policy functions to generate a bid value.

12 . The non-transitory computer readable medium of claim 11 , inputting estimates of a content slot to the first policy function to generate the bid value comprises:

generating a plurality of features related to the content slot;

inputting the plurality of features to a machine learning model to predict the estimate; and

using the estimates generated by the machine learning model as inputs to the first policy function.

13 . The non-transitory computer readable medium of claim 10 , wherein the set of policy functions comprises a plurality of policy functions, each policy function being defined based on an objective provided by the content provider.

14 . The non-transitory computer readable medium of claim 10 , wherein at least one of the policy functions comprises a plurality of states that record past actions and outcomes associated with the policy function.

15 . The non-transitory computer readable medium of claim 10 , wherein the reinforcement learning algorithm updates the set of policy functions through a counterfactual policy estimation.

16 . The non-transitory computer readable medium of claim 10 , wherein the reinforcement learning algorithm is heuristic based, and wherein applying the heuristic based reinforcement learning algorithm comprises:

defining a rule in adjusting a parameter in a policy function;

examining the policy function at a previous state that has a known realized outcome; and

adjusting the parameter based on the rule and the known realized outcome of the previous state.

17 . The non-transitory computer readable medium of claim 10 , wherein the reinforcement learning algorithm is a machine learning based.

18 . The non-transitory computer readable medium of claim 10 , wherein one of the content items in the content presentation campaign is a sponsored item offered on one or more interfaces hosted by the online concierge system.

19 . An online concierge system comprising:

one or more processors; and

memory configured to store code comprising instructions, the instructions, when executed by the one or more processors, cause the one or more processors to:

receive, by an online concierge system from a computer system associated with a content provider, a content presentation campaign that includes one or more objectives set by the content provider, at least one of the objectives defining a desired outcome;

define a set of one or more policy functions that automatically control the content presentation campaign, each policy function comprising one or more parameters and a plurality of states, each state corresponding to a previously used parameter configuration and an associated realized outcome of the content presentation campaign;

monitor a realized outcome of the content presentation campaign that is controlled by the set of policy functions;

store, by the online concierge system, data indicating the realized outcomes and the corresponding states of the policy functions associated with the campaign;

apply a reinforcement learning algorithm that autonomously adjusts one or more parameters of the policy functions using a state-aware counterfactual estimation process that references previously stored states and corresponding outcomes to reduce a difference between the realized outcomes and the desired outcome set by the content provider;

generate an adjusted set of policy functions by the reinforcement learning algorithm; and

use the adjusted set of policy functions to control subsequent content selection operations of the online concierge system, thereby enabling autonomous system adaptation to changing campaign performance.

20 . The online concierge system of claim 19 , wherein using the adjusted set of policy functions comprises inputting estimates of a content slot to a first policy function of the policy functions to generate a bid value.