Low memory content aware image modification
Techniques are disclosed for generating a modified image. A source image at an original, or high, resolution is downsampled to a lower resolution. One or more mappings between different patches, or regions, of pixels within the source image or across several images are generated from the downsampled image using a patch-based sampling algorithm. Such patch-based sampling algorithms may be used find correspondences between patches of one or more images by defining a nearest-neighbor field as a function of offsets for some distance function of the patches. The mapping is then upsampled to a higher resolution for coherent regions and used to identify fill content in the source image for filling patches in other portions of the image or other images. The fill content is derived from regions of the source image corresponding to the coordinates of the source patch in the upsampled mapping.
1. A computer-implemented method comprising:
receiving, by a processor, a source image at an original resolution;
generating, using a patch-based sampling algorithm, a patch coordinate map between a source patch of pixels in a downsampled version of the source image and a target patch of pixels in the downsampled version of the source image, the downsampled version of the source image having a resolution coarser than the original resolution;
upsampling the patch coordinate map to a resolution finer than the resolution of the downsampled version of the source image;
identifying fill content in a region of the original resolution source image corresponding to coordinates of the source patch of pixels in the upsampled patch coordinate map; and
placing the fill content into a target image at the original resolution.
2. The method of claim 1 , wherein the source patch of pixels represents a coherent region of a finer resolution version of the source image reconstructed from the downsampled version of the source image.
3. The method of claim 1 , further comprising combining the source image and the target image to produce an output image.
4. The method of claim 1 , wherein the identifying of the fill content includes at least one of copying and averaging pixel colors from a region of the original resolution source image corresponding to at least a portion of the source patch.
5. The method of claim 1 , further comprising repeating the generating, upsampling and identifying for each of a plurality of overlapping portions of the source image.
6. The method of claim 1 , further comprising storing the mapping in an array.
7. The method of claim 6 , wherein the array has a dimension of up to approximately 33,000 by 33,000.
8. The method of claim 1 , further comprising generating a mask for constraining at least one of: portions of the target image into which the fill content may be placed, and portions of the target image into which the fill content may not be placed.
9. A system comprising:
a storage;
a processor operatively coupled to the storage, the processor configured to execute instructions stored in the storage that when executed cause the processor to carry out a process comprising:
receiving a source image at an original resolution;
generating, using a patch-based sampling algorithm, a patch coordinate map between a source patch of pixels in a downsampled version of the source image and a target patch of pixels in the downsampled version of the source image, the downsampled version of the source image having a resolution coarser than the original resolution;
upsampling the patch coordinate map to a resolution finer than the resolution of the
downsampled version of the source image;
identifying fill content in a region of the original resolution source image corresponding to coordinates of the source patch of pixels in the upsampled patch coordinate map; and
placing the fill content into a target image at the original resolution using an area averaging.
10. The system of claim 9 , wherein the source patch of pixels represents a coherent region of a finer resolution version of the source image reconstructed from the downsampled version of the source image.
11. The system of claim 9 , wherein the process includes combining the source image and the target image to produce an output image.
12. The system of claim 9 , wherein the identifying of the fill content includes at least one of copying and averaging pixel colors from a region of the original resolution source image corresponding to at least a portion of the source patch.
13. The system of claim 9 , wherein the process includes repeating the generating, upsampling and identifying for each of a plurality of overlapping portions of the source image.
14. The system of claim 9 , wherein the process includes storing the mapping in an array.
15. The system of claim 9 , further comprising generating a mask for constraining at least one of: portions of the target image into which the fill content may be placed, and portions of the target image into which the fill content may not be placed.
16. A non-transient computer program product having instructions encoded thereon that when executed by one or more processors cause a process to be carried out, the process comprising:
receiving a source image at an original resolution;
generating using a patch-based sampling algorithm, a patch coordinate map between a source patch of pixels in a downsampled version of the source image and a target patch of pixels in the downsampled version of the source image, the downsampled version of the source image having a resolution coarser than the original resolution;
upsampling, by the processor, the patch coordinate map to a resolution finer than the resolution of the downsampled version of the source image;
identifying, by the processor, fill content in a region of the original resolution source image corresponding to coordinates of the source patch of pixels in the upsampled patch coordinate map; and
placing, by the processor, the fill content into a target image at the original resolution.
17. The computer program product of claim 16 , wherein the source patch of pixels represents a coherent region of a finer resolution version of the source image reconstructed from the downsampled version of the source image.
18. The computer program product of claim 16 , wherein the process includes combining the source image and the target image to produce an output image.
19. The computer program product of claim 16 , wherein the identifying of the fill content includes at least one of copying and averaging pixel colors from a region of the original resolution source image corresponding to at least a portion of the source patch.
20. The computer program product of claim 16 , wherein the process includes repeating the generating, the upsampling and the identifying for each of a plurality of overlapping portions of the source image.