Granted Patent B2

US 12,591,950 · App. 17/584,170 · Granted Mar 31, 2026

Inventors: I-Ming Pao (Palo Alto, CA); Zhe Lin (Fremont, CA); Sarah Stuckey (Petaluma, CA); Jianming Zhang (Campbell, CA); Betty Leong (Los Altos, CA)

Assignee: Adobe Inc.

G06T3/4053G06F18/211G06F18/217G06T3/40G06V10/462G06V10/764G06V10/7747G06V10/82

Abstract

Claims (48)

1 . A system comprising:

one or more memory devices comprising a neural network; and

at least one processor configured to cause the system to:

determine a region of a digital image having a first resolution and comprising a single salient object;

generate, from the region of the digital image comprising the single salient object, a segmentation map of the single salient object having a second resolution corresponding to a predetermined image resolution specification of the neural network by utilizing the neural network to identify pixels of the digital image corresponding to the single salient object in the region of the digital image;

determine whether to perform additional refinement to the segmentation map of the region of the digital image comprising the single salient object;

generate, from a sub-region of the region of the digital image and based on determining to perform additional refinement to the segmentation map, an additional segmentation map of the sub-region of the region having the second resolution by utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region; and

generate, from the digital image, the segmentation map of the region of the digital image, and the additional segmentation map of the sub-region, a refined segmentation map of the digital image having the first resolution that identifies the pixels in the digital image corresponding to the single salient object.

2 . The system of claim 1 , wherein the at least one processor is further configured cause the system to generate the refined segmentation map of the digital image without user input indicating a portion of the digital image corresponding to the single salient object.

3 . The system of claim 1 , wherein the at least one processor is further configured cause the system to generate a modified digital image utilizing the refined segmentation map.

4 . The system of claim 1 , wherein the at least one processor is further configured cause the system to generate a version of the digital image having a reduced-resolution and generate the additional segmentation map and the segmentation map of the region of the digital image comprising the single salient object from the version of the digital image having the reduced-resolution.

5 . The system of claim 4 , wherein the at least one processor is further configured cause the system to upsample the refined segmentation map from the reduced-resolution to a higher resolution.

6 . The system of claim 5 , wherein the at least one processor is further configured to cause the system to upsample the refined segmentation map from the reduced-resolution to the higher resolution by applying at least one of a dense conditional random field filter, a guided filter, or a graph cut filter to the refined segmentation map of the sub-region of the region of the digital image.

7 . The system of claim 1 , wherein utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region comprises:

determining confidence values corresponding to the pixels;

determining that one or more confidence values corresponding to one or more pixels that do not satisfy a threshold condition; and

refining the one or more confidence values utilizing the neural network.

8 . The system of claim 1 , wherein the neural network comprises a convolution neural network trained to identify salient objects in digital images.

9 . A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to perform operations comprising:

generating, utilizing a neural network, an initial segmentation map of a digital image that identifies pixels of the digital image corresponding to a single salient object portrayed in the digital image, the digital image having a first resolution;

generating, from a region of the digital image comprising the single salient object as indicated in the initial segmentation map, a segmentation map of the single salient object portrayed utilizing the neural network to identify pixels of the digital image corresponding to the single salient object in the region of the digital image, the initial segmentation map having a second resolution corresponding to a predetermined image resolution specification of the neural network;

determining whether to perform additional refinement to the segmentation map of the region of the digital image;

generating, from a sub-region of the region of the digital image and based on determining to perform additional refinement to the segmentation map, an additional segmentation map of the sub-region of the region having the second resolution by utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region;

identifying, based on the additional segmentation map, one or more regions of the digital image with confidence values meeting a threshold condition; and

generating a refined segmentation map having the first resolution by reprocessing, utilizing the neural network, the one or more regions of the digital image to refine the confidence values meeting the threshold condition.

10 . The non-transitory computer readable medium of claim 9 , wherein generating the refined segmentation map by reprocessing, utilizing the neural network, the one or more regions of the digital image to refine the confidence values meeting the threshold condition comprises iteratively applying the neural network to regions of the digital image with confidence values meeting the threshold condition as indicated by a prior application of the neural network.

11 . The non-transitory computer readable medium of claim 10 , wherein the threshold condition comprises one or more of:

a predefined number of iterations of applying the neural network;

a predefined duration of time; or

a threshold convergence of confidence values between consecutive iterations.

12 . The non-transitory computer readable medium of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the computing device to perform operations comprising upsampling the refined segmentation map from a reduced-resolution to a higher resolution.

13 . The non-transitory computer readable medium of claim 12 , wherein upsampling the refined segmentation map from the reduced-resolution to the higher resolution comprises applying at least one of a dense conditional random field filter, a guided filter, or a graph cut filter to the refined segmentation map.

14 . The non-transitory computer readable medium of claim 12 , wherein upsampling the refined segmentation map from the reduced-resolution to the higher resolution comprises upsampling based on information from a version of the digital image having the higher resolution.

15 . In a digital medium environment for editing high-resolution digital images, a computer-implemented method comprising:

receiving user input selecting an option to perform object selection on a digital image;

generating, utilizing a neural network, an initial segmentation map of the digital image that identifies pixels of the digital image corresponding to a single salient object portrayed in the digital image, the digital image having a first resolution;

generating, from a region of the digital image comprising the single salient object as indicated in the initial segmentation map, a segmentation map of the single salient object having a second resolution corresponding to a predetermined image resolution specification of the neural network by utilizing the neural network to identify pixels of the region of the digital image corresponding to the single salient object without receiving user input indicating a portion of the digital image corresponding to the single salient object;

determining whether to perform additional refinement to the segmentation map of the region of the digital image;

generating, from a sub-region of the region of the digital image and based on determining to perform additional refinement to the segmentation map, an additional segmentation map of the sub-region having the second resolution by utilizing the neural network to refine identification of the pixels corresponding to the single salient object in the sub-region; and

upsampling the additional segmentation map from the second resolution to the first resolution.

16 . The computer-implemented method of claim 15 , wherein determining whether to perform additional refinement to the segmentation map of the region of the digital image comprises:

analyzing the segmentation map of the single salient object to identify the sub-region of the digital image comprising pixels having confidence values, indicating whether the pixels belong to the single salient object or not, below a threshold value.

17 . The computer-implemented method of claim 15 , further comprising:

generating a version of the digital image with the second resolution by downsampling the digital image; and

utilizing the version of the digital image with the second resolution to generate the segmentation map and the additional segmentation map.

18 . The computer-implemented method of claim 17 , wherein upsampling the additional segmentation map from the second resolution to the first resolution comprises upsampling the additional segmentation map to a resolution of the digital image prior to the downsampling of the digital image.

19 . The computer-implemented method of claim 15 , wherein upsampling the additional segmentation map from the second resolution to the first resolution comprises applying at least one of a dense conditional random field filter, a guided filter, or a graph cut filter to the additional segmentation map.

20 . The computer-implemented method of claim 15 , wherein generating, from the digital image, the segmentation map of the portion of the digital image comprising the single salient object portrayed within the digital image by utilizing the neural network to identify pixels of the digital image corresponding to the single salient object comprises iteratively applying the neural network to regions of the digital image with confidence values meeting a threshold condition as indicated by a prior application of the neural network.

Assignments (2)

Continuity (2)

References Cited (310)