Scene understanding and generation using neural networks

1 . A method performed by one or more computers for generating a numerical representation of a scene, the method comprising:

receiving a plurality of observations characterizing the scene, each observation comprising: (i) an image of the scene, and (ii) data identifying a location of a camera that captured the image;

processing each of the plurality of observations using an observation neural network to generate a respective lower dimensional representation of each of the plurality of observations;

generating the numerical representation of the scene by combining the respective lower dimensional representations of each of the plurality of observations; and

providing the numerical representation of the scene for use in characterizing content of the scene, wherein the observation neural network has been jointly trained with a generator neural network that is configured to:

receive data identifying a new camera location;

process a network input to the generator neural network, the network input comprising: (i) the data identifying the new camera location, and (ii) the numerical representation of the scene; and

generate, by the generator neural network, a network output that comprises a new image of the scene that represents the view of the scene from the camera at the new camera location.

2 . The method of claim 1 , wherein generating the numerical representation of the scene by combining the lower dimensional representations of the plurality of observations comprises:

summing the lower dimensional representations of the plurality of observations.

3 . The method of claim 1 , wherein generating the numerical representation of the scene by combining the lower dimensional representations of the plurality of observations comprises:

processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the scene.

4 . The method of claim 3 , wherein processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the scene comprises:

processing the respective lower dimensional representation corresponding to each of the plurality of observations using a recurrent neural network;

wherein numerical representation of the scene is based on a hidden state of the recurrent neural network after the recurrent neural network has processed the respective lower dimensional representation corresponding to each of the plurality of observations.

5 . The method of claim 1 , wherein for one or more of the plurality of observations, the data identifying the location of the camera that captured the observation defines a three-dimensional position, yaw, and pitch of the camera.

6 . The method of claim 1 , wherein the observation neural network comprises one or more convolutional neural network layers.

7 . The method of claim 1 , wherein providing the numerical representation of the scene for use in characterizing the content of the scene comprises:

computationally rendering the new image of the scene that represents the view of the scene from the camera at the new camera location using the numerical representation of the scene.

8 . The method of claim 7 , wherein computationally rendering the new image of the scene that represents a view of the scene from the camera at the new camera location using the numerical representation of the scene comprises:

receiving data identifying the new camera location;

processing, using the generator neural network, a network input to the generator neural network, the network input comprising: (i) the data identifying the new camera location, and (ii) the numerical representation of the scene; and

generating, by the generator neural network and in response to processing the network input, a network output that comprises the new image of the scene that represents the view of the scene from the camera at the new camera location.

9 . The method of claim 8 , wherein processing the network input using the generator neural network to generate the network output comprises:

at each of a plurality of time steps:

sampling one or more latent variables for the time step; and

updating a hidden state of the generator neural network as of the time step by processing the hidden state, the sampled latent variables, the numerical representation of the scene, and the data identifying the new camera location; and

after a last time step in the plurality of time steps:

generating the new image of the scene from the updated hidden state of the generator neural network.

10 . The method of claim 8 , wherein processing the network input using the generator neural network to generate the network output comprises:

processing the network input, using the generator neural network, to generate respective pixel sufficient statistics for each pixel in the new image of the scene; and

sampling a respective color value for each pixel in the new image of the scene using the pixel sufficient statistics for the pixel.

11 . The method of claim 8 , wherein the generator neural network and the observation neural network have been trained jointly with a posterior neural network that is configured to, during the training, receive a plurality of training observations and a target observation and generate a posterior output that defines a distribution over one or more latent variables.

12 . A method performed by one or more computers for generating a numerical representation of a video, the method comprising:

receiving a plurality of observations characterizing the video, each observation comprising: (i) a video frame of the video, and (ii) data identifying a time stamp of the video frame of the video;

processing each of the plurality of observations using an observation neural network to generate a respective lower dimensional representation of each of the plurality of observations;

generating the numerical representation of the video by combining the respective lower dimensional representations of each of the plurality of observations; and

providing the numerical representation of the video for use in characterizing the video, wherein the observation neural network has been jointly trained with a generator neural network that is configured to:

receive data identifying a new time stamp;

process a network input to the generator neural network, the network input comprising: (i) the data identifying the new time stamp, and (ii) the numerical representation of the video; and

generate, by the generator neural network, a network output that comprises a new video frame at the new time stamp.

13 . The method of claim 12 , wherein generating the numerical representation of the video by combining the lower dimensional representations of the plurality of observations comprises:

summing the lower dimensional representations of the plurality of observations.

14 . The method of claim 12 , wherein generating the numerical representation of the video by combining the lower dimensional representations of the plurality of observations comprises:

processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the video.

15 . The method of claim 14 , wherein processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the video comprises:

processing the respective lower dimensional representation corresponding to each of the plurality of observations using a recurrent neural network;

wherein numerical representation of the video is based on a hidden state of the recurrent neural network after the recurrent neural network has processed the respective lower dimensional representation corresponding to each of the plurality of observations.

16 . The method of claim 12 , wherein for one or more of the plurality of observations, the time stamp of the video frame defines a time at which the video frame was captured.

17 . The method of claim 12 , wherein the observation neural network comprises one or more convolutional neural network layers.

18 . The method of claim 12 , wherein providing the numerical representation of the video for use in characterizing the video comprises:

computationally rendering the new video frame that represents the video frame in the video at the new time stamp using the numerical representation of the video.

19 . The method of claim 18 , wherein computationally rendering a new video frame that represents a video frame in the video at a new time stamp using the numerical representation of the video comprises:

receiving data identifying the new time stamp;

processing, using the generator neural network, a network input to the generator neural network, the network input comprising: (i) the data identifying the new time stamp, and (ii) the numerical representation of the video; and

generating, by the generator neural network and in response to processing the network input, a network output that comprises the new video frame at the new time stamp.

20 . The method of claim 19 , wherein processing the network input using the generator neural network to generate the network output comprises:

at each of a plurality of time steps:

sampling one or more latent variables for the time step; and

updating a hidden state of the generator neural network as of the time step by processing the hidden state, the sampled latent variables, the numerical representation of the video, and the data identifying the new time stamp; and

after a last time step in the plurality of time steps:

generating the new video frame from the updated hidden state of the generator neural network.

21 . The method of claim 19 , wherein processing the network input using the generator neural network to generate the network output comprises:

processing the network input, using the generator neural network, to generate respective pixel sufficient statistics for each pixel in the new video frame; and

sampling a respective color value for each pixel in the new video frame using the pixel sufficient statistics for the pixel.

22 . The method of claim 19 , wherein the generator neural network and the observation neural network have been trained jointly with a posterior neural network that is configured to, during the training, receive a plurality of training observations and a target observation and generate a posterior output that defines a distribution over one or more latent variables.

23 . A method performed by one or more computers for generating a numerical representation of an image, the method comprising:

receiving a plurality of observations characterizing the image, each observation comprising: (i) a crop of the image, and (ii) data identifying a location and size of the crop in the image;

processing each of the plurality of observations using an observation neural network to generate a respective lower dimensional representation of each of the plurality of observations;

generating the numerical representation of the image by combining the respective lower dimensional representations of each of the plurality of observations; and

providing the numerical representation of the image for use in characterizing content of the image, wherein the observation neural network has been jointly trained with a generator neural network that is configured to:

receive data identifying a new crop location and a new crop size of a new crop of the image;

process a network input to the generator neural network, the network input comprising: (i) the data identifying the new crop location and the new crop size, and (ii) the numerical representation of the image; and

generate, by the generator neural network, a network output that comprises the new crop of the image at the new crop location and with the new crop size.

24 . The method of claim 23 , wherein generating the numerical representation of the image by combining the lower dimensional representations of the plurality of observations comprises:

summing the lower dimensional representations of the plurality of observations.

25 . The method of claim 23 , wherein generating the numerical representation of the image by combining the lower dimensional representations of the plurality of observations comprises:

processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the image.

26 . The method of claim 25 , wherein processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the image comprises:

processing the respective lower dimensional representation corresponding to each of the plurality of observations using a recurrent neural network;

wherein numerical representation of the image is based on a hidden state of the recurrent neural network after the recurrent neural network has processed the respective lower dimensional representation corresponding to each of the plurality of observations.

27 . The method of claim 23 , wherein for one or more of the plurality of observations, the data identifying the location and size of the crop in the image includes coordinates of a vertices of a bounding box in the image.

28 . The method of claim 23 , wherein the observation neural network comprises one or more convolutional neural network layers.

29 . The method of claim 23 , wherein providing the numerical representation of the image for use in characterizing the content of the image comprises:

computationally rendering the new crop of the image that represents the crop of the image at the new crop location and having the new crop size using the numerical representation of the image.

30 . The method of claim 29 , wherein computationally rendering the new crop of the image using the numerical representation of the image comprises:

receiving data identifying the new crop location and the new crop size of the new crop;

processing, using the generator neural network, a network input to the generator neural network, the network input comprising: (i) the data identifying the new crop location and the new crop size, and (ii) the numerical representation of the image; and

generating, by the generator neural network and in response to processing the network input, a network output that comprises the new crop of the image at the new crop location and with the new crop size.

31 . The method of claim 30 , wherein processing the network input using the generator neural network to generate the network output comprises:

at each of a plurality of time steps:

sampling one or more latent variables for the time step; and

updating a hidden state of the generator neural network as of the time step by processing the hidden state, the sampled latent variables, the numerical representation of the image, and the data identifying the new crop location; and

after a last time step in the plurality of time steps:

generating the new crop of the image from the updated hidden state of the generator neural network.

32 . The method of claim 30 , wherein processing the network input using the generator neural network to generate the network output comprises:

processing the network input, using the generator neural network, to generate respective pixel sufficient statistics for each pixel in the new crop of the image; and

sampling a respective color value for each pixel in the new crop of the image using the pixel sufficient statistics for the pixel.

33 . The method of claim 30 , wherein the generator neural network and the observation neural network have been trained jointly with a posterior neural network that is configured to, during the training, receive a plurality of training observations and a target observation and generate a posterior output that defines a distribution over one or more latent variables.