Learning observation representations by predicting the future in latent space

1. A method performed by one or more computers, the method comprising:

receiving an input observation; and

processing the input observation using an encoder neural network, in accordance with a plurality of encoder neural network parameters, to generate a latent representation of the input observation;

wherein the encoder neural network has been trained to optimize a loss function, wherein the loss function measures an error between:

(i) a latent representation of an observation that is after a first observation in a sequence of observations, wherein the latent representation of the observation is generated by processing the observation using the encoder neural network; and

(ii) a prediction for the latent representation of the observation;

wherein the prediction for the latent representation of the observation is generated based on a latent representation of a preceding observation that precedes the observation in the sequence of observations,

wherein the latent representation of the preceding observation is generated by processing the preceding observation using the encoder neural network.

2. The method of claim 1 , wherein the prediction for the latent representation of the observation is generated based on: (i) the latent representation of the preceding observation, and (ii) latent representations of one or more observations that are before the preceding observation in the sequence of observations.

3. The method of claim 2 , wherein the encoder neural network has been trained using gradients of the loss function with respect to the plurality of encoder neural network parameters.

4. The method of claim 3 , wherein the gradients of the loss function have been used to adjust values of the plurality of encoder neural network parameters.

5. The method of claim 1 , wherein the sequence of observations comprises a sequence of segments of an audio waveform.

6. The method of claim 1 , wherein the sequence of observations comprises a sequence of regions of a given image or a sequence of respective images.

7. The method of claim 1 , wherein the sequence of observations comprises a sequence of sentences in a natural language.

8. The method of claim 1 , wherein the encoder neural network is configured for use by a reinforcement learning system to generate latent representations of input observations of an environment which are used to select actions to be performed by a reinforcement learning agent to interact with the environment.

9. The method of claim 1 , wherein the encoder neural network comprises one or more convolutional neural network layers, and wherein a latent representation of an observation is a compressed representation of the observation.

10. The method of claim 1 , wherein the prediction for the latent representation of the observation is generated by determining a product between: (i) a context latent representation derived from the latent representation of the preceding observation, and (ii) a prediction parameter matrix.

11. The method of claim 10 , wherein the prediction parameter matrix is selected from among a plurality of prediction parameter matrices based on a number of observations separating: (i) the preceding observation, and (ii) the observation, in the sequence of observations.

12. The method of claim 1 , wherein the loss function comprises a noise contrastive estimation loss function.

13. The method of claim 1 , wherein the loss function measures an error between: (i) the latent representation of the observation, and (ii) the prediction for the latent representation of the observation, based on a similarity between the prediction for the latent representation of the observation and each of: (a) a positive example latent representation, and (b) one or more negative example latent representations;

wherein the positive example latent representation is a latent representation of a particular observation that is representative of the observation;

wherein the negative example latent representation is a latent representation of a specific observation that is not representative of the observation.

14. The method of claim 13 , wherein the loss function includes a ratio of:

the similarity between the prediction for the latent representation of the observation and the positive example latent representation, and

a sum of the respective similarities between the prediction for the latent representation of the observation and: (i) the positive example latent representation, and (ii) each of the one or more negative example latent representations.

15. The method of claim 14 , wherein:

the similarity between the prediction for the latent representation of the observation and the positive example latent representation comprises an exponential of a dot product between the prediction for the latent representation of the observation and the positive example latent representation; and

the similarity between the prediction for the latent representation of the observation and a negative example latent representation comprises an exponential of a dot product between the prediction for the latent representation of the observation and the negative example latent representation.

16. The method of claim 1 , wherein the prediction for the latent representation of the observation is generated by processing a context latent representation;

wherein the context latent representation is generated by processing the latent representation of the preceding observation and latent representations of one or more observations that are before the preceding observation in the sequence of observations using a context neural network in accordance with current values of context neural network parameters.

17. The method of claim 16 , wherein the context neural network has been jointly trained with the encoder neural network.

18. The method of claim 16 , wherein the context neural network comprises one or more recurrent neural network layers.

19. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving an input observation; and

processing the input observation using an encoder neural network, in accordance with a plurality of encoder neural network parameters, to generate a latent representation of the input observation;

wherein the encoder neural network has been trained to optimize a loss function, wherein the loss function measures an error between:

(i) a latent representation of an observation that is after a first observation in a sequence of observations, wherein the latent representation of the observation is generated by processing the observation using the encoder neural network; and

(ii) a prediction for the latent representation of the observation;

wherein the prediction for the latent representation of the observation is generated based on a latent representation of a preceding observation that precedes the observation in the sequence of observations,

wherein the latent representation of the preceding observation is generated by processing the preceding observation using the encoder neural network.

20. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an input observation; and

processing the input observation using an encoder neural network, in accordance with a plurality of encoder neural network parameters, to generate a latent representation of the input observation;

wherein the encoder neural network has been trained to optimize a loss function, wherein the loss function measures an error between:

(i) a latent representation of an observation that is after a first observation in a sequence of observations, wherein the latent representation of the observation is generated by processing the observation using the encoder neural network; and

(ii) a prediction for the latent representation of the observation;

wherein the prediction for the latent representation of the observation is generated based on a latent representation of a preceding observation that precedes the observation in the sequence of observations,

wherein the latent representation of the preceding observation is generated by processing the preceding observation using the encoder neural network.