Method and system for high-resolution image inpainting
Methods and systems for high-resolution image inpainting are disclosed. An original high-resolution image to be inpainted is obtained, as well as an inpainting mask indicating an inside-mask area to be inpainted. The original high-resolution image is down-sampled to obtain a low-resolution image to be inpainted. Using a trained inpainting generator, a low-resolution inpainted image and a set of attention scores are generated from the low-resolution image. The attention scores represent the similarity between inside-mask regions and outside-mask regions. A high-frequency residual image is computed from the original high-resolution image. An aggregated high-frequency residual image is generated using the attention scores, including high-frequency residual information for the inside-mask area. A high-resolution inpainted image is outputted by combining the aggregated high-frequency residual image and a low-frequency inpainted image generated from the low-resolution inpainted image.
1. A method for high-resolution image inpainting, comprising:
obtaining an original high-resolution image to be inpainted, and an inpainting mask indicating an inside-mask area to be inpainted;
down-sampling the original high-resolution image to obtain a low-resolution image to be inpainted;
generating, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image and a set of attention scores, the set of attention scores representing a similarity between regions of the inside-mask area and regions of an outside-mask area;
computing, from the original high-resolution image, a high-frequency residual image;
generating, using the set of attention scores, an aggregated high-frequency residual image including computed high-frequency residual information for at least the inside-mask area; and
outputting, as a high-resolution inpainted image, a combination of at least the aggregated high-frequency residual image and a low-frequency inpainted image, the low-frequency inpainted image being generated from the low-resolution inpainted image.
2. The method of claim 1 , wherein the inpainting generator comprises a coarse network and a refine network, the coarse network generating a coarse inpainting output from the low-resolution image, the coarse inpainting output and the low-resolution image being provided as input to the refine network to output the low-resolution inpainted image and the set of attention scores.
3. The method of claim 2 , wherein the set of attention scores is computed based on a highest-level feature map of the refine network.
4. The method of claim 2 , wherein the set of attention scores is used as aggregation weights to compute multiple contextualized feature maps for respective layers of the refine network.
5. The method of claim 1 , wherein the inpainting generator is a trained generator network that was trained using low-resolution images as training data.
6. The method of claim 5 , wherein the training data includes randomly generated inpainting masks to be applied to the low-resolution images.
7. The method of claim 1 , wherein the aggregated high-frequency residual image includes high-frequency residual information computed for the inside-mask area, using weighted aggregation of high-frequency residual information from the outside-mask area, the set of attention scores being used as aggregation weights.
8. The method of claim 7 , wherein generating the aggregated high-frequency residual image comprises:
calculating, for each given region of the inside-mask area, high-frequency residual information for the given region of the inside-mask area using a weighted aggregation of high-frequency residual information from every region of the outside-mask area, the set of attention scores being used as the aggregation weights.
9. The method of claim 7 , wherein the aggregated high-frequency residual image includes high-frequency residual information for only the inside-mask area, wherein the aggregated high-frequency image is combined with the low-frequency inpainted image for only the inside-mask area, and wherein the combination of the aggregated high-frequency image and the low-frequency inpainted image is further combined with the outside-mask area of the original high-resolution image, to output the high-resolution inpainted image.
10. The method of claim 7 , wherein the aggregated high-frequency residual image includes high-frequency residual information for the inside-mask area, and also includes high-frequency residual information for the outside-mask area, wherein the aggregated high-frequency residual image is combined with the low-frequency inpainted image to output the high-resolution inpainted image.
11. A device for high-resolution image inpainting, comprising:
a processor configured to execute instructions to cause the device to:
obtain an original high-resolution image to be inpainted, and an inpainting mask indicating an inside-mask area to be inpainted;
down-sample the original high-resolution image to obtain a low-resolution image to be inpainted;
generate, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image and a set of attention scores, the set of attention scores representing a similarity between regions of the inside-mask area and regions of an outside-mask area;
compute, from the original high-resolution image, a high-frequency residual image;
generate, using the set of attention scores, an aggregated high-frequency residual image including computed high-frequency residual information for at least the inside-mask area; and
output, as a high-resolution inpainted image, a combination of at least the aggregated high-frequency residual image and a low-frequency inpainted image, the low-frequency inpainted image being generated from the low-resolution inpainted image.
12. The device of claim 11 , wherein the inpainting generator comprises a coarse network and a refine network, the coarse network generating a coarse inpainting output from the low-resolution image, the coarse inpainting output and the low-resolution image being provided as input to the refine network to output the low-resolution inpainted image and the set of attention scores.
13. The device of claim 12 , wherein the set of attention scores is computed based on a highest-level feature map of the refine network.
14. The device of claim 12 , wherein the set of attention scores is used as aggregation weights to compute multiple contextualized feature maps for respective layers of the refine network.
15. The device of claim 11 , wherein the inpainting generator is a trained generator network that was trained using low-resolution images as training data, the training data also including randomly generated inpainting masks to be applied to the low-resolution images during training.
16. The device of claim 11 , wherein the aggregated high-frequency residual image includes high-frequency residual information computed for the inside-mask area, using weighted aggregation of high-frequency residual information from the outside-mask area, the set of attention scores being used as aggregation weights.
17. The device of claim 16 , wherein the processor is configured to execute instructions to cause the device to generate the aggregated high-frequency residual image by:
calculating, for each given region of the inside-mask area, high-frequency residual information for the given region of the inside-mask area using a weighted aggregation of high-frequency residual information from every region of the outside-mask area, the set of attention scores being used as the aggregation weights.
18. The device of claim 16 , wherein the aggregated high-frequency residual image includes high-frequency residual information for only the inside-mask area, wherein the aggregated high-frequency image is combined with the low-frequency inpainted image for only the inside-mask area, and wherein the combination of the aggregated high-frequency image and the low-frequency inpainted image is further combined with the outside-mask area of the original high-resolution image, to output the high-resolution inpainted image.
19. The device of claim 16 , wherein the aggregated high-frequency residual image includes high-frequency residual information for the inside-mask area, and also includes high-frequency residual information for the outside-mask area, wherein the aggregated high-frequency residual image is combined with the low-frequency inpainted image to output the high-resolution inpainted image.
20. A non-transitory computer-readable medium having instructions encoded thereon, wherein the instructions, when executed by a processor of a device, cause the device to:
obtain an original high-resolution image to be inpainted, and an inpainting mask indicating an inside-mask area to be inpainted;
down-sample the original high-resolution image to obtain a low-resolution image to be inpainted;
generate, from the low-resolution image using a trained inpainting generator, a low-resolution inpainted image and a set of attention scores, the set of attention scores representing a similarity between regions of the inside-mask area and regions of an outside-mask area;
compute, from the original high-resolution image, a high-frequency residual image;
generate, using the set of attention scores, an aggregated high-frequency residual image including computed high-frequency residual information for at least the inside-mask area; and
output, as a high-resolution inpainted image, a combination of at least the aggregated high-frequency residual image and a low-frequency inpainted image, the low-frequency inpainted image being generated from the low-resolution inpainted image.