Training action selection neural networks using hindsight modelling

Inventors: Arthur Clement Guez (London, GB); Fabio Viola (Montréal, CA); Theophane Guillaume Weber (London, GB); Lars Buesing (Letchworth Garden City, GB); Nicolas Manfred Otto Heess (London, GB)

Assignee: GDM Holding LLC

G06N3/084 G06N3/08 G06V10/82

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12688427
App. No.: 17/763,901
Granted: Jul 21, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A reinforcement learning method and system that selects actions to be performed by a reinforcement learning agent interacting with an environment. A causal model is implemented by a hindsight model neural network and trained using hindsight i.e. using future environment state trajectories. As the method and system does not have access to this future information when selecting an action, the hindsight model neural network is used to train a model neural network which is conditioned on data from current observations, which learns to predict an output of the hindsight model neural network.

Claims (34)

1 . A computer implemented method of reinforcement learning, comprising:

training an action selection neural network system to select actions to be performed by an agent in an environment for performing a task,

wherein the action selection neural network system is configured to receive input data from i) an observation characterizing a current state of the environment and ii) an output of a model neural network, and to process the input data in accordance with action selection neural network system parameters to generate an action selection output for selecting the actions to be performed by the agent; and

wherein the model neural network is configured to receive an input derived from the observation characterizing the current state of the environment and the output of the model neural network characterizes a predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state;

wherein the method further comprises: training a hindsight model neural network having an output characterizing a state trajectory comprising a series of k states of the environment starting from a state of the environment at a time step t, by processing data from one or more observations characterizing the state of the environment at the time step t and at a series of k subsequent time steps the training the hindsight model comprising:

processing, using a hindsight value neural network, the output of the hindsight model neural network and data from the observation characterizing the state of the environment at the time step t, to generate an estimated hindsight value or state-action value for the state of the environment at the time step t, and backpropagating gradients of an objective function dependent upon a difference between the estimated hindsight value or state-action value for the state of the environment at the time step t and a training goal for the time step t to update parameters of the hindsight value neural network and the parameters of the hindsight model neural network; and

training the model neural network to generate an output that approximates the output of the hindsight model neural network generated from the observation characterizing the current state of the environment and the one or more observations characterizing the state of the environment at the series of k subsequent time steps after the time step t.

2 . The method of claim 1 wherein the action selection neural network system includes a state value neural network for selecting, or learning to select, the actions to be performed by the agent.

3 . The method of claim 2 wherein training the action selection neural network system comprises backpropagating gradients of an objective function dependent upon a difference between a state value or state-action value for the current state of the environment determined using the state value neural network and an estimated return or state-action value for the current state of the environment.

4 . The method of claim 2 wherein the action selection neural network system has parameters in common with the model neural network.

5 . The method of claim 1 further comprising providing, as an input to the action selection neural network system and to the hindsight value neural network, an internal state of one or more recurrent neural network (RNN) layers which receive data from the observations characterizing the state of the environment.

6 . The method of claim 1 further comprising providing, as an input to the hindsight model neural network and to the model neural network, an internal state of one or more recurrent neural network (RNN) layers which receive data from the observations characterizing the state of the environment.

7 . The method of claim 1 wherein the training goal for the time step t comprises a state value or state-action value target for the time step t.

8 . The method of claim 1 wherein the training goal for the time step t comprises an estimated return for the time step t.

9 . The method of claim 1 wherein training the output of the model neural network to approximate the output of the hindsight model neural network comprises backpropagating gradients of an objective function dependent upon a difference between the output of the hindsight model neural network characterizing the state trajectory and the output of the model neural network characterizing the predicted state trajectory.

10 . The method of claim 1 wherein the output of the hindsight model neural network characterizing the state trajectory and the output of the model neural network characterizing the predicted state trajectory each comprise features of a reduced dimensionality representation of one or more observations of the environment.

11 . The method of claim 1 wherein the output of the hindsight model neural network characterizing the state trajectory and the output of the model neural network characterizing the predicted state trajectory each have a dimensionality of less than their input, less than 20, or less than 10.

12 . The method of claim 1 wherein k is less than 20.

13 . The method of claim 1 further comprising maintaining a memory that stores data representing trajectories generated as a result of interaction of the agent with the environment, each trajectory comprising data at each of a series of time steps identifying at least an observation characterizing a state of the environment and a series of subsequent observations characterizing subsequent states of the environment for training the hindsight model neural network.

14 . A reinforcement learning neural network system, comprising: an action selection neural network system to select actions to be performed by an agent in an environment for performing a task,

wherein the model neural network is configured to receive an input from the observation characterizing the current state of the environment and the output of the model neural network characterizes a predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state; wherein the system is configured to:

train a hindsight model neural network having an output characterizing a state trajectory comprising a series of k states of the environment starting from a state of the environment at a time step t, by processing observations characterizing the state of the environment at the time step t and at a series of k subsequent time steps training the hindsight model comprising:

train the model neural network to generate, an output that approximates the output of the hindsight model neural network generated from the observation characterizing the current state of the environment and the one or more observations characterizing the state of the environment at the series of k subsequent time steps after the time step t.

15 . A neural network computer system for performing a task, comprising: an action selection neural network system to select actions to be performed by an agent in an environment for performing a task,

a model neural network, wherein the model neural network is configured to receive an input from the observation characterizing the current state of the environment and the output of the model neural network characterizes a predicted state trajectory comprising a series of k predicted future states of the environment starting from the current state, wherein the model neural network has been trained to generate, an output that approximates an output of a hindsight model neural network that the hindsight model neural network would have generated from the observation characterizing the current state of the environment and one or more observations characterizing the state of the environment at the series of k subsequent time steps after a time step t, wherein the hindsight model neural network has been trained by processing the one or more observations characterizing the state of the environment at the time step t and at a series of k subsequent time steps, the training the hindsight model comprising:

16 . The system of claim 14 wherein the action selection neural network system includes a state value neural network for selecting, or learning to select, the actions to be performed by the agent.

17 . The system of claim 16 wherein training the action selection neural network system comprises backpropagating gradients of an objective function dependent upon a difference between a state value or state-action value for the current state of the environment determined using the state value neural network and an estimated return or state-action value for the current state of the environment.

18 . The system of claim 16 wherein the action selection neural network system has parameters in common with the model neural network.

19 . The system of claim 14 further comprising providing, as an input to the action selection neural network system and to the hindsight value neural network, an internal state of one or more recurrent neural network (RNN) layers which receive data from the observations characterizing the state of the environment.

20 . The system of claim 14 further comprising providing, as an input to the hindsight model neural network and to the model neural network, an internal state of one or more recurrent neural network (RNN) layers which receive data from the observations characterizing the state of the environment.