Image editing for small masks

1 . A computer-implemented method, comprising:

receiving a first source image;

receiving first mask data comprising a first mask region that identifies first pixels in the first source image;

generating first cropped image data comprising the first pixels and second pixels at least partially surrounding the first pixels in the first source image;

generating a second source image by upsampling the first cropped image data;

generating an upsampled mask region by upsampling the first mask region by a same factor as the upsampling the first cropped image data;

receiving an inpainting prompt;

generating, by a diffusion-based inpainting model, a first output image comprising the second source image inpainted in a region corresponding to the upsampled mask data, wherein the second source image is inpainted based on the inpainting prompt;

generating downsampled output data by downsampling the first output image by the same factor; and

generating a second output image by replacing the first pixels corresponding to the first mask region in the first source image with corresponding inpainted pixels in the downsampled output data.

2 . The computer-implemented method of claim 1 , further comprising:

determining a first area of the first mask region;

determining a second area of the first source image;

determining a ratio of the first area to the second area; and

determining that the ratio is below a threshold ratio, wherein the generating the first cropped image data and the generating the second source image data by upsampling are performed based on the ratio being below the threshold ratio.

3 . The computer-implemented method of claim 1 , wherein the first mask data comprises a binary mask that masks out pixels in the first mask data that have a first binary value and maintains pixels of the first mask data that have a second binary value.

4 . The computer-implemented method of claim 1 , wherein the first cropped image data comprises an RGB image.

5 . A method comprising:

receiving a first source image;

receiving first mask region identifying first pixels of the first source image;

determining a second source image comprising the first pixels of the first source image and second pixels of the first source image that at least partially surround the first pixels;

generating upsampled image data by upsampling the second source image;

generating upsampled mask data by upsampling image data of the first mask region;

generating, by a diffusion-based inpainting model, a first inpainted image comprising the upsampled image data inpainted in a region corresponding to the upsampled mask data; and

generating a first downsampled image by downsampling the first inpainted image.

6 . The method of claim 5 , further comprising generating the first downsampled image by downsampling the first inpainted image to a resolution of the first source image.

7 . The method of claim 6 , further comprising:

determining second pixels of the first downsampled image corresponding to the first pixels of the first mask region; and

generating an output image by replacing the first pixels of the first source image with the second pixels of the first downsampled image.

8 . The method of claim 5 , wherein first mask data comprises a binary mask that is configured to mask out pixels in the first mask data that have a first binary value and maintain pixels of the first mask data that have a second binary value, wherein pixels of the first mask region have the first binary value.

9 . The method of claim 5 , wherein the second source image comprises an RGB image that is cropped from the first source image.

10 . The method of claim 5 , further comprising:

determining a first number of the first pixels in the first mask region;

determining a second number of pixels in the first source image;

determining a ratio of the first number to the second number; and

determining that the ratio is below a threshold ratio, wherein the determining the second source image and the generating the upsampled image data are performed based on the ratio being below the threshold ratio.

11 . The method of claim 5 , further comprising:

receiving a first reference image, wherein the generating the first inpainted image comprises inpainting a representation of an entity in the first reference image in the region of the upsampled image data corresponding to the upsampled mask data.

12 . The method of claim 5 , further comprising determining that a first size of the first mask region is below a threshold size relative to a second size of the first source image, wherein the upsampled image data and the upsampled mask data is generated based at least in part on the first size of the first mask region being below the threshold size relative to the second size of the first source image.

13 . A system comprising:

at least one processor; and

non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to:

receive a first source image;

receive first mask region identifying first pixels of the first source image;

determine a second source image comprising the first pixels of the first source image and second pixels of the first source image that at least partially surround the first pixels;

generate upsampled image data by upsampling the second source image;

generate upsampled mask data by upsampling image data of the first mask region;

generate, by a diffusion-based inpainting model, a first inpainted image comprising the upsampled image data inpainted in a region corresponding to the upsampled mask data; and

generate a first downsampled image by downsampling the first inpainted image.

14 . The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

generate the first downsampled image by downsampling the first inpainted image to a resolution of the first source image.

15 . The system of claim 14 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine second pixels of the first downsampled image corresponding to the first pixels of the first mask region; and

generate an output image by replacing the first pixels of the first source image with the second pixels of the first downsampled image.

16 . The system of claim 13 , wherein first mask data comprises a binary mask that is configured to mask out pixels in the first mask data that have a first binary value and maintain pixels of the first mask data that have a second binary value, wherein pixels of the first mask region have the first binary value.

17 . The system of claim 13 , wherein the second source image comprises an RGB image that is cropped from the first source image.

18 . The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine a first number of the first pixels in the first mask region;

determine a second number of pixels in the first source image;

determine a ratio of the first number to the second number; and

determine that the ratio is below a threshold ratio, wherein the determining the second source image and the generating the upsampled image data is performed based on the ratio being below the threshold ratio.

19 . The system of claim 13 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

receive a first reference image, wherein the generating the first inpainted image comprises inpainting a representation of an entity in the first reference image in the region of the upsampled image data corresponding to the upsampled mask data.

20 . The system of claim 19 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:

determine that a first size of the first mask region is below a threshold size relative to a second size of the first source image, wherein the upsampled image data and the upsampled mask data is generated based at least in part on the first size of the first mask region being below the threshold size relative to the second size of the first source image.