Generating predictions characterizing images using glimpses extracted from the images
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using recurrent attention. One of the methods includes determining a location in the first image; extracting a glimpse from the first image using the location; generating a glimpse representation of the extracted glimpse; processing the glimpse representation using a recurrent neural network to update a current internal state of the recurrent neural network to generate a new internal state; processing the new internal state to select a location in a next image in the image sequence after the first image; and processing the new internal state to select an action from a predetermined set of possible actions.
1. A method performed by one or more computers, the method comprising:
generating a prediction characterizing one or more images, comprising:
extracting a plurality of glimpses from the one or more images, comprising, at each iteration preceding a last iteration of a plurality of iterations:
receiving data identifying a current location in a current image of the one or more images;
extracting a current glimpse from the current image at the current location;
processing a network input derived from the current glimpse extracted from the current image at the current location using a location neural network, in accordance with values of a set of location neural network parameters, to generate a next location in a next image of the one or more images; and
providing the next location in the next image for processing at a next iteration of the plurality of iterations; and
generating the prediction characterizing the one or more images using one or more of the plurality of glimpses extracted from the one or more images.
2. The method of claim 1 , wherein the one or more images comprise a plurality of images defining a video.
3. The method of claim 1 , wherein at each iteration, extracting the current glimpse from the current image at the current location comprises:
extracting a plurality of patches from the current image, each patch being centered at the current location in the current image; and
combining the plurality of patches to generate the current glimpse.
4. The method of claim 3 , wherein each of the plurality of patches has a distinct size, and wherein each of the plurality of patches has a distinct resolution.
5. The method of claim 3 , wherein combining the plurality of patches to generate the current glimpse comprises:
re-scaling the patches so that each patch has a same size; and
concatenating the re-scaled patches to generate the current glimpse.
6. The method of claim 1 , wherein at each iteration, the network input to the location neural network is generated by operations comprising:
generating a current glimpse representation of the current glimpse extracted from the current image at the current location, comprising:
processing the current glimpse and the current location in the current image using a glimpse neural network to generate the glimpse representation; and
generating the network input to the location neural network based at least in part on the current glimpse representation.
7. The method of claim 6 , wherein processing the current glimpse and the current location in the current image using the glimpse neural network to generate the current glimpse representation comprises:
processing the current glimpse using one or more first neural network layers to generate an initial representation of the current glimpse;
processing the current location in the current image using one or more second neural network layers to generate an initial representation of the current location in the current image; and
processing the initial representation of the current glimpse and the initial representation of the current location in the current image using one or more third neural network layers to generate the current glimpse representation.
8. The method of claim 1 , wherein processing the network input derived from the current glimpse extracted from the current image at the current location using the location neural network to generate the next location in the next image of the one or more images comprises:
processing the network input using the location neural network to generate a distribution parameter; and
stochastically selecting the next location from a distribution of possible locations that is parameterized by the distribution parameter.
9. The method of claim 1 , wherein the location neural network is trained by a reinforcement learning optimization procedure.
10. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
generating a prediction characterizing one or more images, comprising:
extracting a plurality of glimpses from the one or more images, comprising, at each iteration preceding a last iteration of a plurality of iterations:
receiving data identifying a current location in a current image of the one or more images;
extracting a current glimpse from the current image at the current location;
processing a network input derived from the current glimpse extracted from the current image at the current location using a location neural network, in accordance with values of a set of location neural network parameters, to generate a next location in a next image of the one or more images; and
providing the next location in the next image for processing at a next iteration of the plurality of iterations; and
generating the prediction characterizing the one or more images using one or more of the plurality of glimpses extracted from the one or more images.
11. The system of claim 10 , wherein the one or more images comprise a plurality of images defining a video.
12. The system of claim 10 , wherein at each iteration, extracting the current glimpse from the current image at the current location comprises:
extracting a plurality of patches from the current image, each patch being centered at the current location in the current image; and
combining the plurality of patches to generate the current glimpse.
13. The system of claim 12 , wherein each of the plurality of patches has a distinct size, and wherein each of the plurality of patches has a distinct resolution.
14. The system of claim 12 , wherein combining the plurality of patches to generate the current glimpse comprises:
re-scaling the patches so that each patch has a same size; and
concatenating the re-scaled patches to generate the current glimpse.
15. The system of claim 10 , wherein at each iteration, the network input to the location neural network is generated by operations comprising:
generating a current glimpse representation of the current glimpse extracted from the current image at the current location, comprising:
processing the current glimpse and the current location in the current image using a glimpse neural network to generate the glimpse representation; and
generating the network input to the location neural network based at least in part on the current glimpse representation.
16. The system of claim 15 , wherein processing the current glimpse and the current location in the current image using the glimpse neural network to generate the current glimpse representation comprises:
processing the current glimpse using one or more first neural network layers to generate an initial representation of the current glimpse;
processing the current location in the current image using one or more second neural network layers to generate an initial representation of the current location in the current image; and
processing the initial representation of the current glimpse and the initial representation of the current location in the current image using one or more third neural network layers to generate the current glimpse representation.
17. The system of claim 10 , wherein processing the network input derived from the current glimpse extracted from the current image at the current location using the location neural network to generate the next location in the next image of the one or more images comprises:
processing the network input using the location neural network to generate a distribution parameter; and
stochastically selecting the next location from a distribution of possible locations that is parameterized by the distribution parameter.
18. The system of claim 10 , wherein the location neural network is trained by a reinforcement learning optimization procedure.
19. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
generating a prediction characterizing one or more images, comprising:
extracting a plurality of glimpses from the one or more images, comprising, at each iteration preceding a last iteration of a plurality of iterations:
receiving data identifying a current location in a current image of the one or more images;
extracting a current glimpse from the current image at the current location;
processing a network input derived from the current glimpse extracted from the current image at the current location using a location neural network, in accordance with values of a set of location neural network parameters, to generate a next location in a next image of the one or more images; and
providing the next location in the next image for processing at a next iteration of the plurality of iterations; and
generating the prediction characterizing the one or more images using one or more of the plurality of glimpses extracted from the one or more images.
20. The one or more non-transitory computer storage media of claim 19 , wherein the one or more images comprise a plurality of images defining a video.