Granted Patent B2

US 12,536,625 · App. 18/058,601 · Granted Jan 27, 2026

Inventors: Qing Liu (Santa Clara, CA); Zhe Lin (Fremont, CA); Luis Figueroa (San Jose, CA); Scott Cohen (Sunnyvale, CA)

Assignee: Adobe Inc.

G06T5/77G06T5/70G06T7/11G06T7/194G06T2200/24G06T2207/20084

Abstract

Claims (67)

1 . A computer-implemented method comprising:

providing, for display within a graphical user interface of a client device, a digital image displaying a plurality of objects;

generating, utilizing a segmentation neural network and without user input, object masks for objects of the plurality of objects;

identifying an object to remove from the digital image;

determining foreground abutting an object mask for the object to remove;

determining background abutting the object mask for the object to remove;

generating an expanded object mask by:

expanding the object mask into the foreground abutting the object mask by a first amount; and

expanding the object mask into the background abutting the object mask by a second amount that differs from the first amount; and

inpainting a hole corresponding to the expanded object mask utilizing an inpainting neural network.

2 . The computer-implemented method of claim 1 , wherein expanding the object mask into the background abutting the object mask by the second amount that differs from the first amount comprises expanding the object mask into the background abutting the object mask by the second amount that is greater than the first amount.

3 . The computer-implemented method of claim 1 , wherein expanding the object mask into the foreground abutting the object mask by the first amount comprises:

expanding the object mask into the foreground abutting the object mask by the second amount; and

reducing an expansion of the object mask into the foreground so that the expansion corresponds to the first amount.

4 . The computer-implemented method of claim 3 ,

further comprising determining that expanding the object mask into the foreground abutting the object mask by the second amount overlaps with at least one object mask of at least one other object portrayed in the digital image,

wherein reducing the expansion of the object mask into the foreground so that the expansion corresponds to the first amount comprises removing overlapping pixels from the expanded object mask.

5 . The computer-implemented method of claim 1 , further comprising:

identifying an additional object to remove from the digital image;

expanding an additional object mask for the additional object by expanding around the object mask by the second amount; and

modifying the digital image by deleting the expanded additional object mask.

6 . The computer-implemented method of claim 1 , wherein inpainting the hole corresponding to the expanded object mask utilizing the inpainting neural network comprises:

generating, before deleting the expanded object mask and without user input, a content fill for the expanded object mask utilizing the inpainting neural network;

placing, before deleting the expanded object mask and without user input, the content fill behind the expanded object mask; and

exposing the content fill upon deleting the expanded object mask.

7 . The computer-implemented method of claim 6 , wherein generating the content fill for the expanded object mask utilizing the inpainting neural network comprises:

generating, utilizing the inpainting neural network, foreground pixels for the foreground abutting the object mask corresponding to the first amount; and

generating, utilizing the inpainting neural network, background pixels for the background butting the object mask corresponding to the second amount.

8 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

providing, for display within a graphical user interface of a client device, a digital image displaying a plurality of objects;

generating, utilizing a segmentation neural network and without user input, object masks for objects of the plurality of objects;

identifying an object to remove from the digital image;

generating an expanded object mask by expanding an object mask for the object into areas not occupied by other object masks;

deleting the expanded object mask; and

inpainting a hole corresponding to the expanded object mask utilizing an inpainting neural network.

9 . The non-transitory computer-readable medium of claim 8 , wherein expanding the object mask into the areas not occupied by the other object masks comprises:

generating a combined object mask corresponding to other objects from the plurality of objects; and

expanding the object mask into areas not occupied by the combined object mask corresponding to the other objects.

10 . The non-transitory computer-readable medium of claim 9 , wherein:

generating, utilizing the segmentation neural network, the object masks for the objects of the plurality of objects comprises generating, utilizing the segmentation neural network, the object mask for the object and the other object masks for the other objects; and

generating the combined object mask corresponding to the other objects comprises generating the combined object mask based on a union of the other object masks for the other objects.

11 . The non-transitory computer-readable medium of claim 9 , wherein expanding the object mask into the areas not occupied by the other object masks comprises:

expanding the object mask of the object into portions of the digital image that abut the object mask by a set amount of pixels;

detecting a region of overlap between the expanded object mask for the object and the combined object mask corresponding to the other objects; and

removing the region of overlap from the expanded object mask for the object.

12 . The non-transitory computer-readable medium of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

detecting, utilizing a shadow detection neural network, a shadow for the object within the digital image; and

deleting the shadow from the digital image along with the expanded object mask.

13 . The non-transitory computer-readable medium of claim 8 , wherein identifying the object to remove from the digital image comprises detecting a user selection of the object via the graphical user interface of the client device.

14 . The non-transitory computer-readable medium of claim 8 , wherein identifying the object to remove from the digital image comprises classifying the object as a distracting object utilizing a distractor detection neural network.

15 . The non-transitory computer-readable medium of claim 8 , wherein inpainting the hole corresponding to the expanded object mask utilizing the inpainting neural network comprises generating, utilizing the inpainting neural network, a content fill for the expanded object mask for the object, the content fill covering the areas not covered by the other object masks.

16 . A system comprising:

at least one memory device comprising a segmentation neural network; and

at least one processor configured to cause the system to:

receive, from a client device, a digital image portraying a plurality of objects having a first object overlapping a second object;

generate, utilizing the segmentation neural network and without user input, object masks for the plurality of objects;

generate an expanded object mask for the first object by:

expanding an object mask for the first object by an expansion amount;

detecting a region of overlap between the expanded object mask for the first object and an object mask for the second object; and

removing the region of overlap from the expanded object mask for the first object; and

modify the digital image by deleting the first object from the digital image using the expanded object mask for the first object.

17 . The system of claim 16 , wherein removing the region of overlap from the expanded object mask for the first object comprises removing a portion of the expanded object mask corresponding to a subset of the region of overlap.

18 . The system of claim 16 , wherein the at least one processor is further configured to cause the system to:

generate, utilizing an inpainting neural network and without user input, a content fill for the expanded object mask for the first object; and

inpainting a hole that remains in the digital image after deleting the expanded object mask utilizing the content fill generated via the inpainting neural network.

19 . The system of claim 16 , wherein the at least one processor is further configured to cause the system to identify the first object for deletion by receiving a user selection to delete the first object or classifying the first object as a distracting object.

20 . The system of claim 16 , wherein detecting the region of overlap between the expanded object mask for the first object and the object mask for the second object comprises detecting the region of overlap between the expanded object mask for the first object and a combined object mask that corresponds to the second object and at least one third object portrayed in the digital image.

Assignments (1)

Continuity (1)

References Cited (259)