Hierarchical planning through goal-conditioned offline reinforcement learning
A method and system for controlling a device includes training a low-level policy to form a trained low-level policy and a low-level value function to form a trained goal conditioned value function, wherein training is performed using a static data set using goal conditioned episodes, training a high-level goal planner having high level goals having high-level sub-goals corresponding to a plurality of future time steps using the low-level value function to maximize a cumulative reward over the sub-goals for the plurality of future time steps so that the sub-goals are reachable by the low-level policy, obtaining an observation of a device, and generating an executable action using the low-level policy and the high-level goal planner and operating the device with the executable action.
1 . A method comprising:
training a low-level policy to form a trained low-level policy and a low-level value function to form a trained goal conditioned value function, wherein training is performed using a static data set using goal conditioned episodes, said low-level value function comprising a reinforcement learning policy;
training a high-level goal planner having high level goals having high-level sub-goals corresponding to a plurality of future time steps using the low-level value function and using a conditional variable auto encoder to model transitions from one sub-goal to a next sub-goal so that sub-goal sequences can be sequentially reconstructed to maximize a cumulative reward over the sub-goals for the plurality of future time steps so that the sub-goals are reachable by the low-level policy;
obtaining an observation of a device;
generating an executable action using the low-level policy, the high-level goal planner and the observation; and
operating the device with the executable action.
2 . The method of claim 1 further comprising, prior to training the low-level policy, forming the goal-conditioned episodes using hindsight experience replay.
3 . The method of claim 1 further comprising training the low-level policy using goal-conditioned episodes and out-of distribution goals by a perturbed goal sampling process.
4 . The method of claim 1 wherein training the high-level goal planner comprises using Markov decision processes between sub-goals of consecutive sub-goals.
5 . The method of claim 1 wherein the high-level goal planner is model-based comprising solving an optimization problem based on a low-level value function for a long-term strategy.
6 . The method of claim 1 wherein generating the executable action comprises generating the executable action based on maximizing the value function.
7 . The method of claim 1 wherein training the high-level planner comprises selecting one of the plurality of sub-goals of a sequence of subgoals at a second time step subsequent to a first time step.
8 . The method of claim 1 wherein training the high-level goal planner comprises generating the high-level sub-goals comprising reachable goals based on a reward.
9 . The method of claim 1 wherein training the low-level policy comprises training the low-level policy and value function.
10 . The method of claim 1 wherein the sub-goals correspond to positions of an automotive vehicle and wherein operating the device comprises controlling a steering wheel angle of a vehicle, an acceleration of the vehicle, or both.
11 . A system comprising a computer-readable medium storing computer-executable instructions that, when executed by a processor, configure the processor to:
train a low-level policy to form a trained low-level policy and a low-level value function to form a trained goal conditioned value function, wherein training is performed using a static data set using goal conditioned episodes, said low-level value function comprising a reinforcement learning policy;
train a high-level goal planner having high level goals having high-level sub-goals corresponding to a plurality of future time steps using the low-level value function and using a conditional variable auto encoder to model transitions from one sub-goal to a next sub-goal so that sub-goal sequences can be sequentially reconstructed to maximize a cumulative reward over the sub-goals for the plurality of future time steps so that the sub-goals are reachable by the low-level policy;
obtain an observation of a device;
generate an executable action using the observation, the low-level policy and the high-level goal planner; and
operate the device with the executable action.
12 . The system of claim 11 further comprising, prior to the processor configured to train the low-level policy, form goal-conditioned episodes from the static data set using hindsight experience replay.
13 . The system of claim 11 further comprising training the low-level policy using goal-conditioned episodes and out-of distribution goals by a perturbed goal sampling process.
14 . The system of claim 11 wherein the processor is configured to train the high-level goal planner using Markov decision processes between sub-goals of consecutive sub-goals.
15 . The system of claim 11 wherein the high-level goal planner is model-based solves an optimization problem based on a low-level value function for a long-term strategy.
16 . The system of claim 11 wherein the processor is further configured to generate the executable actions based on maximizing the value function.
17 . The system of claim 11 wherein the processor is further configured to train the high-level planner by selecting one of the plurality of sub-goals of a sequence of subgoals at a second time step subsequent to a first time step.
18 . The system of claim 11 wherein the processor is further configured to train the high-level goal planner by generating the high-level sub-goals comprising reachable goals based on a reward.
19 . The system of claim 11 wherein the processor is further configured to train the low-level policy and value function.
20 . The system of claim 11 wherein the sub-goals correspond to positions of an automotive vehicle and the processor is further configured to control a steering wheel angle of a vehicle, an acceleration of the vehicle, or both.