Unsupervised control using learned rewards

1. A system for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the system comprising:

an action selection subsystem that is configured to, at each of a plurality of time steps:

receive an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment;

process the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an action selection neural network to generate an output comprising an action selection policy; and

select an action from the predetermined set of actions to be performed by the agent using the action selection policy;

a reward subsystem that is configured to, at each of a plurality of time steps:

receive an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment;

process the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an embedding neural network, wherein the embedding neural network is configured to process an observation to generate an embedded representation of the observation; and

generate a reward using the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment; and

a training subsystem that is configured to:

train the action selection neural network based on the rewards generated by the reward subsystem using reinforcement learning techniques.

2. The system of claim 1 , wherein the reward subsystem is further configured to:

obtain one or more decoy observations, wherein each decoy observation characterizes a past state of the environment;

for each decoy observation, generate an embedded representation of the decoy observation by processing the decoy observation using the embedding neural network;

determine a similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using: (i) the embedded representation of the observation characterizing the current state of the environment, (ii) the embedded representation of the observation characterizing the goal state of the environment, and (iii) the embedded representations of the decoy observations.

3. The system of claim 2 , wherein determining the similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment comprises:

for each decoy observation, determining a similarity metric between the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the decoy observation;

determining a similarity metric between the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment; and

determining the similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using the determined similarity metrics.

4. The system of claim 3 , wherein the similarity metric is an inner product.

5. The system of claim 2 , wherein generating the reward comprises:

generating the reward based on the determined similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment.

6. The system of claim 1 , wherein generating the reward comprises:

determining the reward based on a similarity between the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment.

7. The system of claim 2 , wherein the training subsystem is further configured to:

determine a gradient of a loss function that depends on the determined similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment; and

adjust current values of embedding neural network parameters using the gradient.

8. The system of claim 1 , wherein the reward subsystem is configured to generate a predetermined reward if a current time step is not a goal time step.

9. The system of claim 1 ,

wherein the system further comprises a goal buffer comprising a plurality of past observations, and

wherein the training subsystem is configured to determine the observation characterizing the goal state of the environment by sampling from the goal buffer.

10. The system of claim 9 , wherein the training subsystem is configured to update the goal buffer by, at each of a plurality of time steps, replacing a randomly selected observation from the goal buffer by a current observation with a predetermined probability.

11. The system of claim 9 , wherein the training subsystem is configured to update the goal buffer by, at each of a plurality of time steps:

randomly selecting a candidate observation from the goal buffer;

determining whether to replace the candidate observation by a current observation based on: (i) an aggregate similarity measure between the current observation and the observations included in the goal buffer, and (ii) an aggregate similarity measure between the candidate observation and the observations included in the goal buffer.

12. The system of claim 1 , wherein the action selection neural network is a Q neural network and the action selection policy comprises a respective Q value for each action in the predetermined set of actions.

13. The system of claim 12 , wherein selecting an action comprises determining an action with a highest Q value.

14. The system of claim 1 , wherein selecting an action comprises selecting the action in accordance with an exploration policy.

15. The system of claim 1 , wherein the action selection neural network is further configured to process a periodic representation of the current time step.

16. The system of claim 1 , wherein the training subsystem is configured to train the action selection neural network and the embedding neural network using experience replay.

17. The system of claim 16 , wherein training the action selection neural network and the embedding neural network using experience replay comprises training the action selection neural network and the embedding neural network using hindsight experience replay.

18. The system of claim 1 , wherein embedding neural network and the action selection neural network share one or more parameter values.

19. A method performed by one or more data processing apparatus for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the method comprising:

using an action selection subsystem to perform operations comprising, at each of a plurality of time steps:

receiving an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment;

processing the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an action selection neural network to generate an output comprising an action selection policy; and

selecting an action from the predetermined set of actions to be performed by the agent using the action selection policy;

using a reward subsystem to perform operations comprising, at each of a plurality of time steps:

receiving an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment;

processing the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an embedding neural network, wherein the embedding neural network is configured to process an observation to generate an embedded representation of the observation; and

generating a reward using the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment; and

using a training subsystem to train the action selection neural network based on the rewards generated using the reward subsystem using reinforcement learning techniques.

20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions, the operations comprising:

using an action selection subsystem to perform operations comprising, at each of a plurality of time steps:

receiving an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment;

processing the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an action selection neural network to generate an output comprising an action selection policy; and

selecting an action from the predetermined set of actions to be performed by the agent using the action selection policy;

using a reward subsystem to perform operations comprising, at each of a plurality of time steps:

receiving an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment;

processing the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an embedding neural network, wherein the embedding neural network is configured to process an observation to generate an embedded representation of the observation; and

generating a reward using the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment; and

using a training subsystem to train the action selection neural network based on the rewards generated using the reward subsystem using reinforcement learning techniques.