Enabling classical planning in interactive environments with incomplete models

1 . A computer-implemented method in a model-based reinforcement learning (RL) system with logic states, the computer-implemented method comprising:

switching an agent between a first mode and a second mode, the first mode being a probabilistic planning mode and the second mode being an information gathering mode, wherein

in the probabilistic planning mode, the agent:

computes a predictive state representation based on historical observations and historical actions taken by the agent; and

scores action candidates based on planning with the predictive state representation such that one or more first actions with resolved plans and with confidence to achieve a goal state are scored higher than one or more second actions without the resolved plans, wherein the action candidates include the one or more first actions and the one or more second actions; and

in the information gathering mode, the agent:

scores the action candidates based on a Q function of a value of expected information to be gathered from a given pair of state and action; and

performs one or more information gathering actions from the scored action candidates based on an intrinsic reward of each action of the one or more information gathering actions, wherein

among the scored action candidates, the one or more information gathering actions provide a maximum increase in knowledge about a current state of an interactive environment,

the intrinsic reward of each action of the one or more information gathering actions corresponds to an amount of increase in the knowledge based on the performing of a respective action of the one or more information gathering actions, and

the agent interacts with the interactive environment.

2 . The computer-implemented method according to claim 1 , wherein

the Q function is learned from trajectories in the interactive environment, and

an increase in a belief state certainty associated with the agent is given as the intrinsic reward.

3 . The computer-implemented method according to claim 2 , wherein the intrinsic reward is an information gain reward based on a local measure of information gain between the current state and a next state of the interactive environment.

4 . The computer-implemented method of claim 3 , wherein the information gain reward uses a number of novel facts in the next state compared to the current state.

5 . The computer-implemented method according to claim 1 , wherein, in a situation where a path cannot be found in a classical planning problem, the agent switches from the probabilistic planning mode to the information gathering mode.

6 . The computer-implemented method according to claim 1 , wherein

the agent is provided with the goal state, and

the agent is in the interactive environment with partial observability without a path to reach the goal state.

7 . The computer-implemented method according to claim 1 , wherein

the agent is provided with the goal state, and

the agent is in an incomplete model that precludes a path to reach the goal state.

8 . The computer-implemented method according to claim 1 , further comprising learning or updating, with a learning algorithm, action models within the probabilistic planning mode using one or more previous interactions with the interactive environment.

9 . The computer-implemented method according to claim 1 , wherein an initial action model of the interactive environment is given to the agent by one or more domain experts.

10 . A computer-implemented method for planning in an interactive environment with partial observability and incomplete models, the computer-implemented method comprising:

producing, with a classical planning algorithm, a path from a given initial state to a goal state based on a model of a world being known to an agent;

learning, with a Q-learning algorithm, from rewards obtained during an interaction, action candidates gathered from a value function that scores state-action pairs, wherein the action candidates are scored based on the value function;

performing, using an information gathering policy, one or more information gathering actions from the scored action candidates based on an intrinsic reward of each action of the one or more information gathering actions, wherein

among the scored action candidates, the one or more information gathering actions provide a maximum increase in knowledge about a current state of the interactive environment,

the intrinsic reward of each action of the one or more information gathering actions corresponds to an amount of increase in the knowledge based on the performing of a respective action of the one or more information gathering actions, and

the agent interacts with the interactive environment; and

switching, with an online replanning framework, between using the classical planning algorithm and using the information gathering policy based on a current model of the world.

11 . The computer-implemented method of claim 10 , further comprising learning or updating, with a learning algorithm, action models within the classical planning algorithm using one or more previous interactions with the interactive environment.

12 . The computer-implemented method according to claim 10 , wherein an initial action model of the interactive environment is given to the agent by one or more domain experts.

13 . The computer-implemented method of claim 10 , wherein

the Q-learning algorithm is learned from trajectories in the interactive environment; and

an increase in a belief state certainty associated with the agent is given as the intrinsic reward.

14 . The computer-implemented method according to claim 13 , wherein the intrinsic reward is an information gain reward defined based on a local measure of information gain between the current state and a next state of the interactive environment.

15 . The computer-implemented method of claim 14 , wherein the information gain reward uses a number of novel facts in the next state compared to a number of facts known in the current state.

16 . A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method for planning in an interactive environment with partial observability and incomplete models, the method comprising:

producing, with a classical planning algorithm, a path from a given initial state to a goal state based on a model of a world being known to an agent;

learning, with a Q-learning algorithm, from rewards obtained during an interaction, action candidates gathered from a value function that scores state-action pairs, wherein the action candidates are scored based on the value function;

performing, using an information gathering policy, one or more information gathering actions from the scored action candidates based on an intrinsic reward of each action of the one or more information gathering actions, wherein

among the scored action candidates, the one or more information gathering actions provide a maximum increase in knowledge about a current state of the interactive environment,

the intrinsic reward of each action of the one or more information gathering actions corresponds to an amount of increase in the knowledge based on the performing of a respective action of the one or more information gathering actions, and

the agent interacts with the interactive environment; and

switching, with an online replanning framework, between using the classical planning algorithm and using the information gathering policy based on a current model of the world.

17 . The non-transitory computer readable storage medium of claim 16 , the method further comprising learning or updating, with a learning algorithm, action models within the classical planning algorithm using one or more previous interactions with the interactive environment.

18 . The non-transitory computer readable storage medium of claim 16 , the method further comprising giving an initial action model of the interactive environment to the agent by one or more domain experts.

19 . The non-transitory computer readable storage medium of claim 16 , wherein

the Q-learning algorithm is learned from trajectories in the interactive environment, and

an increase in a belief state certainty associated with the agent is given as the intrinsic reward.