Image processing of an environment to select an action to be performed by an agent interacting with the environment

1. A method for processing one or more images of an environment to select an action to be performed by an agent interacting with the environment, the method comprising:

at each iteration of a plurality of iterations:

receiving a current image for the iteration, wherein the current image is included in the one or more images;

determining a location in the current image, comprising:

determining the location based on an output of a location neural network for the current iteration if the current iteration is after a first iteration in the plurality of iterations;

extracting a glimpse from the current image using the location;

updating a current internal state of a recurrent neural network using the glimpse extracted from the current image to generate a new internal state, comprising:

generating a glimpse representation of the extracted glimpse, and

processing the glimpse representation using the recurrent neural network to update the current internal state of the recurrent neural network to generate a new internal state;

processing, using the location neural network, the new internal state of the recurrent neural network generated using the glimpse extracted from the image to generate an output of the location neural network for a next iteration in the plurality of iterations; and

at a last iteration of the plurality of iterations, selecting the action to be performed by the agent, including:

processing, using an action neural network, the new internal state of the recurrent neural network at the last iteration to generate a respective action score for each action in a set of actions; and

selecting the action to be performed by the agent using the action scores;

wherein the location neural network, the recurrent neural network, and the action neural network have been trained by an end-to-end optimization procedure.

2. The method of claim 1 , wherein the agent comprises a robot.

3. The method of claim 2 , wherein the robot comprises an autonomous vehicle, aircraft, or watercraft.

4. The method of claim 2 , wherein the images of the environment characterize current surroundings of the robot.

5. The method of claim 2 , wherein the set of actions comprises navigational actions for controlling motion of the robot in the environment.

6. The method of claim 1 , wherein each image of the one or more images is a same image.

7. The method of claim 1 , wherein selecting the action to be performed by the agent using the action scores comprises:

selecting a highest-scoring action according to the prediction generated using the action neural network.

8. The method of claim 1 , wherein extracting the glimpse from the current image comprises:

extracting a plurality of patches from the current image, each patch being centered at the location in the current image; and

combining the patches to generate the glimpse.

9. The method of claim 8 , wherein each of the plurality of patches has a distinct size, and wherein each of the plurality of patches has a distinct resolution.

10. The method of claim 9 , wherein combining the patches to generate the glimpse comprises:

re-scaling the patches so that each patch has a same size; and

concatenating the re-scaled patches to generate the glimpse.

11. The method of claim 1 , wherein generating a glimpse representation of the extracted glimpse comprises:

processing the extracted glimpse and the location in the current image using a glimpse neural network to generate the glimpse representation.

12. The method of claim 11 , wherein processing the extracted glimpse and the location in the current image using the glimpse neural network to generate the glimpse representation comprises:

processing the extracted glimpse using one or more first neural network layers to generate an initial representation of the extracted glimpse;

processing the location in the current image using one or more second neural network layers to generate an initial representation of the location in the current image; and

processing the initial representation of the extracted glimpse and the initial representation of the location in the current image using one or more third neural network layers to generate the glimpse representation.

13. The method of claim 1 , wherein processing the new internal state to generate an output of the location neural network comprises:

processing the new internal state using the location neural network to generate a distribution parameter; and

stochastically selecting a location from a distribution of possible locations that is parameterized by the distribution parameter.

14. The method of claim 1 , wherein the recurrent neural network is a long short term memory (LSTM) neural network.

15. The method of claim 1 , wherein the optimization procedure comprises a reinforcement learning optimization procedure.

16. The method of claim 15 , wherein the reinforcement learning optimization procedure comprises training the location neural network, the recurrent neural network, and the action neural network to maximize a measure of total reward received as a result of actions performed by the agent in the environment.

17. The method of claim 16 , wherein rewards received as a result of actions performed by the agent in the environment are based on progress of the agent toward accomplishing goals in the environment.

18. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for processing one or more images of an environment to select an action to be performed by an agent interacting with the environment, the operations comprising:

at each iteration of a plurality of iterations:

receiving a current image for the iteration, wherein the current image is included in the one or more images;

determining a location in the current image, comprising:

determining the location based on an output of a location neural network for the current iteration if the current iteration is after a first iteration in the plurality of iterations;

extracting a glimpse from the current image using the location;

updating a current internal state of a recurrent neural network using the glimpse extracted from the current image to generate a new internal state, comprising:

generating a glimpse representation of the extracted glimpse, and

processing the glimpse representation using the recurrent neural network to update the current internal state of the recurrent neural network to generate a new internal state;

processing, using the location neural network, the new internal state of the recurrent neural network generated using the glimpse extracted from the image to generate an output of the location neural network for a next iteration in the plurality of iterations; and

at a last iteration of the plurality of iterations, selecting the action to be performed by the agent, including:

processing, using an action neural network, the new internal state of the recurrent neural network at the last iteration to generate a respective action score for each action in a set of actions; and

selecting the action to be performed by the agent using the action scores;

wherein the location neural network, the recurrent neural network, and the action neural network have been trained by an end-to-end optimization procedure.

19. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for processing one or more images of an environment to select an action to be performed by an agent interacting with the environment, the operations comprising:

at each iteration of a plurality of iterations:

receiving a current image for the iteration, wherein the current image is included in the one or more images;

determining a location in the current image, comprising:

determining the location based on an output of a location neural network for the current iteration if the current iteration is after a first iteration in the plurality of iterations;

extracting a glimpse from the current image using the location;

updating a current internal state of a recurrent neural network using the glimpse extracted from the current image to generate a new internal state, comprising:

generating a glimpse representation of the extracted glimpse, and

processing the glimpse representation using the recurrent neural network to update the current internal state of the recurrent neural network to generate a new internal state;

processing, using the location neural network, the new internal state of the recurrent neural network generated using the glimpse extracted from the image to generate an output of the location neural network for a next iteration in the plurality of iterations; and

at a last iteration of the plurality of iterations, selecting the action to be performed by the agent, including:

processing, using an action neural network, the new internal state of the recurrent neural network at the last iteration to generate a respective action score for each action in a set of actions; and

selecting the action to be performed by the agent using the action scores;

wherein the location neural network, the recurrent neural network, and the action neural network have been trained by an end-to-end optimization procedure.

20. The non-transitory computer storage media of claim 19 , wherein the agent comprises a robot.