Spatial transformer modules
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using an image processing neural network system that includes a spatial transformer module. One of the methods includes receiving an input feature map derived from the one or more input images, and applying a spatial transformation to the input feature map to generate a transformed feature map, comprising: processing the input feature map to generate spatial transformation parameters for the spatial transformation, and sampling from the input feature map in accordance with the spatial transformation parameters to generate the transformed feature map.
1. An image processing neural network system implemented by one or more computers, wherein the image processing neural network system is configured to receive one or more input images and to process the one or more input images to generate a neural network output from the one or more input images, the image processing neural network system comprising:
a spatial transformer module, wherein the spatial transformer module is configured to perform operations comprising:
receiving an input feature map derived from the one or more input images, and
applying a spatial transformation to the input feature map to generate a transformed feature map, comprising:
processing the input feature map to generate, based on the input feature map, spatial transformation parameters that define the spatial transformation to be applied to the input feature map, and
sampling from the input feature map in accordance with the spatial transformation parameters generated based on the input feature map to generate the transformed feature map.
2. The image processing neural network system of claim 1 , the operations further comprising:
providing the transformed feature map as input to another component of the image processing neural network system.
3. The image processing neural network system of claim 1 , wherein the input feature map is an output generated by another component of the image processing neural network system.
4. The image processing neural network system of claim 3 , wherein the other component of the image processing neural network system is another spatial transformer module.
5. The image processing neural network system of claim 4 , wherein the other spatial transformer module performs a different type of spatial transformation than the spatial transformer module.
6. The image processing neural network system of claim 3 , wherein the other component of the image processing neural network system is a neural network layer.
7. The image processing neural network system of claim 1 , wherein the input feature map is one of the one or more input images.
8. The image processing neural network system of claim 1 ,
wherein the spatial transformer module comprises:
a localisation subnetwork comprising one or more neural network layers, wherein the localisation subnetwork is configured to process the input feature map to generate the spatial transformation parameters in accordance with current values of a set of parameters of the localisation subnetwork, and
wherein processing the input feature map to generate the spatial transformation parameters comprises processing the input feature map using the localisation subnetwork.
9. The image processing neural network system of claim 1 , wherein sampling from the input feature map in accordance with the spatial transformation parameters to generate the transformed feature map comprises:
generating, using the transformation parameters, a sampling grid that defines, for each of a plurality of locations in the transformed feature map, how a value of the location should be derived from values in the input feature map; and
sampling from the input feature map in accordance with the sampling grid to generate the transformed feature map.
10. The image processing neural network system of claim 1 , wherein the sampling mechanism is differentiable.
11. The image processing neural network system of claim 10 , wherein the spatial transformer module has been trained using backpropagation during training of the image processing neural network system.
12. The image processing neural network system of claim 1 , wherein the transformed feature map has the same dimensions as the input feature map.
13. The image processing neural network system of claim 1 , wherein the transformed feature map has different dimensions from the input feature map.
14. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement an image processing neural network that is configured to receive one or more input images and to process the one or more input images to generate a neural network output from the one or more input images, the image processing neural network comprising:
a spatial transformer module, wherein the spatial transformer module is configured to perform operations comprising:
receiving an input feature map derived from the one or more input images, and
applying a spatial transformation to the input feature map to generate a transformed feature map, comprising:
processing the input feature map to generate, based on the input feature map, spatial transformation parameters that define the spatial transformation to be applied to the input feature map, and
sampling from the input feature map in accordance with the spatial transformation parameters generated based on the input feature map to generate the transformed feature map.
15. The computer storage media of claim 14 , the operations further comprising:
providing the transformed feature map as input to another component of the image processing neural network system.
16. The computer storage media of claim 14 , wherein the input feature map is an output generated by another component of the image processing neural network system.
17. The computer storage media of claim 14 , wherein the input feature map is one of the one or more input images.
18. The computer storage media of claim 14 ,
wherein the spatial transformer module comprises:
a localisation subnetwork comprising one or more neural network layers, wherein the localisation subnetwork is configured to process the input feature map to generate the spatial transformation parameters in accordance with current values of a set of parameters of the localisation subnetwork, and
wherein processing the input feature map to generate the spatial transformation parameters comprises processing the input feature map using the localisation subnetwork.
19. The computer storage media of claim 14 , wherein sampling from the input feature map in accordance with the spatial transformation parameters to generate the transformed feature map comprises:
generating, using the transformation parameters, a sampling grid that defines, for each of a plurality of locations in the transformed feature map, how a value of the location should be derived from values in the input feature map; and
sampling from the input feature map in accordance with the sampling grid to generate the transformed feature map.
20. A method comprising:
training an image processing neural network on training images,
wherein the image processing neural network is configured to receive one or more input images and to process the one or more input images to generate a neural network output from the one or more input images,
wherein the image processing neural network comprises a spatial transformer module,
wherein the spatial transformer module is configured to perform operations comprising:
receiving an input feature map derived from the one or more input images, and applying a spatial transformation to the input feature map to generate a transformed feature map, comprising: processing the input feature map to generate, based on the input feature map, spatial transformation parameters that define the spatial transformation to be applied to the input feature map, and sampling from the input feature map in accordance with the spatial transformation parameters generated based on the input feature map to generate the transformed feature map, and
wherein training the image processing neural network comprises:
backpropagating gradients computed from errors between the neural network outputs generated by the image processing neural network for the training images and known outputs for the training images to train the spatial transformer module.
21. The method of claim 20 , wherein the spatial transformer module comprises a localisation subnetwork comprising one or more neural network layers, wherein the localisation subnetwork is configured to process the input feature map to generate the spatial transformation parameters in accordance with current values of a set of parameters of the localisation subnetwork.
22. The method of claim 21 , wherein the sampling mechanism is differentiable, and wherein backpropagating the gradients comprises backpropagating the gradients through the sampling mechanism to adjust the values of the parameters of the localisation subnetwork.