Imagination-based agent neural networks
A neural network system is proposed. The neural network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which receives a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include a rollout encoder to encode the features, and an output stage to receive data derived from the rollout embedding and to output action policy data for identifying an action based on the current observation.
1. A method for controlling an agent interacting with an environment to perform a task, the method comprising:
receiving a current observation characterizing a current state of the environment at a current time step;
generating, from the current observation, and using a model of the environment, an imagination code, wherein the imagination code is data that characterizes each of one or more predicted future trajectories for the agent starting from the current time step;
processing the current observation using a model-free reinforcement learning neural network to generate a model-free output, wherein the model-free reinforcement learning neural network does not comprise a model of the environment;
generating, from the imagination code and the model-free output, action policy data that defines an action policy for controlling the agent in response to the current observation; and
selecting, using the action policy, an action to be performed by the agent in response to the current observation.
2. The method of claim 1 , wherein generating the imagination code comprises:
generating, from the current observation and using the model of the environment, the one or more predicted future trajectories;
generating, for each of the one or more predicted future trajectories, a respective rollout embedding that represents a summary of the predicted future trajectory; and
generating the imagination code from the respective rollout embeddings.
3. The method of claim 2 wherein the model of the environment is a neural environment model that is configured to receive an input comprising an input observation or a history of observations, and a current action and to predict at least a subsequent observation in response; and wherein generating each of the one or more predicted future trajectories comprises, for each of the future time steps:
selecting, using a policy module, an action for a preceding time step,
generating an input for the future time step from the selected action for the preceding time step and an observation at the preceding time step, and
generating, using the neural environment model, a predicted observation for the future time step.
4. The method of claim 2 wherein generating each respective rollout embedding comprises processing the predicted future trajectory using a sequential state generation neural network.
5. The method of claim 1 , wherein each predicted future trajectory comprises a respective predicted observation at each of a plurality of future time steps that are after the current time step, the respective predicted observation at each of the plurality of future time steps characterizing a predicted state of the environment at the future time step.
6. The method of claim 4 , wherein each predicted future trajectory further comprises, for each of the future time steps, one or more of: a predicted action, a predicted reward, or a predicted sequence termination signal.
7. The method of claim 1 wherein the model of the environment is a learned model of the environment.
8. The method of claim 1 , wherein generating, from the imagination code and the model-free output, action policy data that defines an action policy for controlling the agent in response to the current observation, comprises:
processing the imagination code and the model-free output using a reinforcement learning neural network to generate the action policy data.
9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for controlling an agent interacting with an environment to perform a task, the operations comprising:
receiving a current observation characterizing a current state of the environment at a current time step;
generating, from the current observation, and using a model of the environment, an imagination code, wherein the imagination code is data that characterizes each of one or more predicted future trajectories for the agent starting from the current time step;
processing the current observation using a model-free reinforcement learning neural network to generate a model-free output, wherein the model-free reinforcement learning neural network does not comprise a model of the environment;
generating, from the imagination code and the model-free output, action policy data that defines an action policy for controlling the agent in response to the current observation; and
selecting, using the action policy, an action to be performed by the agent in response to the current observation.
10. The system of claim 9 , wherein generating the imagination code comprises:
generating, from the current observation and using the model of the environment, the one or more predicted future trajectories;
generating, for each of the one or more predicted future trajectories, a respective rollout embedding that represents a summary of the predicted future trajectory; and
generating the imagination code from the respective rollout embeddings.
11. The system of claim 10 wherein the model of the environment is a neural environment model that is configured to receive an input comprising an input observation or a history of observations, and a current action and to predict at least a subsequent observation in response; and wherein generating each of the one or more predicted future trajectories comprises, for each of the future time steps:
selecting, using a policy module, an action for a preceding time step,
generating an input for the future time step from the selected action for the preceding time step and an observation at the preceding time step, and
generating, using the neural environment model, a predicted observation for the future time step.
12. The system of claim 10 wherein generating each respective rollout embedding comprises processing the predicted future trajectory using a sequential state generation neural network.
13. The system of claim 9 , wherein each predicted future trajectory comprises a respective predicted observation at each of a plurality of future time steps that are after the current time step, the respective predicted observation at each of the plurality of future time steps characterizing a predicted state of the environment at the future time step.
14. The system of claim 13 , wherein each predicted future trajectory further comprises, for each of the future time steps, one or more of: a predicted action, a predicted reward, or a predicted sequence termination signal.
15. The system of claim 9 wherein the model of the environment is a learned model of the environment.
16. The system of claim 9 , wherein generating, from the imagination code and the model-free output, action policy data that defines an action policy for controlling the agent in response to the current observation, comprises:
processing the imagination code and the model-free output using a reinforcement learning neural network to generate the action policy data.
17. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for controlling an agent interacting with an environment to perform a task, the operations comprising:
receiving a current observation characterizing a current state of the environment at a current time step;
generating, from the current observation, and using a model of the environment, an imagination code, wherein the imagination code is data that characterizes each of one or more predicted future trajectories for the agent starting from the current time step;
processing the current observation using a model-free reinforcement learning neural network to generate a model-free output, wherein the model-free reinforcement learning neural network does not comprise a model of the environment;
generating, from the imagination code and the model-free output, action policy data that defines an action policy for controlling the agent in response to the current observation; and
selecting, using the action policy, an action to be performed by the agent in response to the current observation.
18. The non-transitory computer-readable storage media of claim 17 , wherein generating the imagination code comprises:
generating, from the current observation and using the model of the environment, the one or more predicted future trajectories;
generating, for each of the one or more predicted future trajectories, a respective rollout embedding that represents a summary of the predicted future trajectory; and
generating the imagination code from the respective rollout embeddings.
19. The non-transitory computer-readable storage media of claim 18 wherein the model of the environment is a neural environment model that is configured to receive an input comprising an input observation or a history of observations, and a current action and to predict at least a subsequent observation in response; and wherein generating each of the one or more predicted future trajectories comprises, for each of the future time steps:
selecting, using a policy module, an action for a preceding time step,
generating an input for the future time step from the selected action for the preceding time step and an observation at the preceding time step, and
generating, using the neural environment model, a predicted observation for the future time step.
20. The non-transitory computer-readable storage media of claim 18 wherein generating each respective rollout embedding comprises processing the predicted future trajectory using a sequential state generation neural network.