Systems and methods for generating object detection labels using foveated image magnification for autonomous driving
Systems and methods for processing high resolution images are disclosed. The methods include generating a saliency map of a received high-resolution image using a saliency model. The saliency map includes a saliency value associated with each of a plurality of pixels of the high-resolution image. The method then includes using the saliency map for generating an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image, and implementing an image warp for converting the high-resolution image to the warped image using the inverse transformation function. The warped image is a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image.
1. A method of processing a high-resolution image, the method comprising, by a processor:
receiving a high-resolution image;
generating, using a saliency model, a saliency map of the high-resolution image, the saliency map comprising a saliency value associated with each of a plurality of pixels of the high-resolution image;
generating, using the saliency map, an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image;
implementing, using the inverse transformation function, an image warp for converting the high-resolution image to the warped image, the warped image being a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image; and
saving the warped image to a data store.
2. The method of claim 1 , further comprising:
generating, using an object detection model, one or more bounding box predictions in a frame of reference of the warped image; and
transforming, using the inverse transformation function, first coordinates of the one or more bounding box predictions in the warped image to second coordinates of the one or more bounding box predictions in a frame of reference of the high-resolution image.
3. The method of claim 2 , further comprising using the second coordinates of the one or more bounding box predictions for controlling navigation of an autonomous vehicle.
4. The method of claim 1 , further comprising generating the saliency model based on one or more bounding box predictions in at least one prior frame of a video stream, the high-resolution image being a part of the video stream and is captured after the at least one prior frame.
5. The method of claim 1 , further comprising generating the saliency model based on one or more bounding box predictions in a dataset-wide prior comprising a training dataset.
6. The method of claim 1 , wherein the at least one region having the higher resolution in the warped image has a high likelihood of including an object of interest.
7. The method of claim 1 , further comprising reducing a resolution of the one or more other regions of the warped image.
8. The method of claim 1 , wherein implementing, using the inverse transformation function, the image warp for converting the high-resolution image to the warped image comprises for each of a plurality of pixels of the warped image:
finding an input pixel in the high-resolution image; and
bi-linearly interpolating that pixel's intensity or color from one or more pixels in the high-resolution image adjacent the input pixel.
9. The method of claim 1 , wherein the inverse transformation function is a differentiable function that is trained using backpropagation.
10. The method of claim 1 , further comprising introducing symmetries about each of a plurality of edges of the saliency map for cropping regularization of the warped image.
11. A system for processing a high-resolution image, the system comprising:
a processor; and
a non-transitory computer readable medium comprising programming instructions that when executed by the processor, will cause the processor to:
receive a high-resolution image;
generate, using a saliency model, a saliency map of the high-resolution image, the saliency map comprising a saliency value associated with each of a plurality of pixels of the high-resolution image;
generate, using the saliency map, an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image;
implement, using the inverse transformation function, an image warp for converting the high-resolution image to the warped image, the warped image being a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image; and
save the warped image to a data store.
12. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to:
generate, using an object detection model, one or more bounding box predictions in a frame of reference of the warped image; and
transform, using the inverse transformation function, first coordinates of the one or more bounding box predictions in the warped image to second coordinates of the one or more bounding box predictions in a frame of reference of the high-resolution image.
13. The system of claim 12 , further comprising programming instructions that when executed by the processor, will cause the processor to use the second coordinates of the one or more bounding box predictions for controlling navigation of an autonomous vehicle.
14. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to generate the saliency model based on one or more bounding box predictions in at least one prior frame of a video stream, the high-resolution image being a part of the video stream and is captured after the at least one prior frame.
15. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to generate the saliency model based on one or more bounding box predictions in a dataset-wide prior comprising a training dataset.
16. The system of claim 11 , wherein the at least one region having the higher resolution in the warped image has a high likelihood of including an object of interest.
17. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to reduce a resolution of the one or more other regions of the warped image.
18. The system of claim 11 , wherein the programming instructions that when executed by the processor, will cause the processor to implement, using the inverse transformation function, the image warp for converting the high-resolution image to the warped image further comprise programming instructions to cause the processor to, for each of a plurality of pixels of the warped image:
find an input pixel in the high-resolution image; and
bi-linearly interpolate that pixel's intensity or color from one or more pixels in the high-resolution image adjacent the input pixel.
19. The system of claim 11 , wherein the inverse transformation function is a differentiable function that is trained using backpropagation.
20. A non-transitory computer program product for processing a high-resolution image, the computer program product comprising a memory that stores programming instructions that are configured to cause a processor to:
receive a high-resolution image;
generate, using a saliency model, a saliency map of the high-resolution image, the saliency map comprising a saliency value associated with each of a plurality of pixels of the high-resolution image;
generate, using the saliency map, an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image;
implement, using the inverse transformation function, an image warp for converting the high-resolution image to the warped image, the warped image being a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image; and
save the warped image to a data store.