Generating image mattes without trimap segmentations via a multi-branch neural network
Methods, systems, and non-transitory computer readable storage media are disclosed for generating image mattes for detected objects in digital images without trimap segmentation via a multi-branch neural network. The disclosed system utilizes a first neural network branch of a generative neural network to extract a coarse semantic mask from a digital image. The disclosed system utilizes a second neural network branch of the generative neural network to extract a detail mask based on the coarse semantic mask. Additionally, the disclosed system utilizes a third neural network branch of the generative neural network to fuse the coarse semantic mask and the detail mask to generate an image matte. In one or more embodiments, the disclosed system also utilizes a refinement neural network to generate a final image matte by refining selected portions of the image matte generated by the generative neural network.
1. A method comprising:
extracting, by at least one processor utilizing a first neural network branch of a generative neural network, a coarse semantic mask from a digital image;
extracting, by the at least one processor utilizing a second neural network branch of the generative neural network, a detail mask from the coarse semantic mask and initial image features extracted from the digital image;
generating, by the at least one processor utilizing a third neural network branch of the generative neural network, an intermediate image matte by fusing the coarse semantic mask and the detail mask; and
generating a final image matte by:
dividing the intermediate image matte into a plurality of patches, having a predetermined size and corresponding to different coordinates of the intermediate image matte;
determining a selected subset of patches of the plurality of patches based on error values corresponding to the plurality of patches from an error map comprising predicted absolute residual errors of pixels of the intermediate image matte; and
refining the selected subset of patches of the intermediate image matte utilizing a refinement neural network.
2. The method of claim 1 , wherein extracting the coarse semantic mask comprises extracting, utilizing the first neural network branch, the coarse semantic mask and intermediate image features from a downsampled image extracted from the digital image.
3. The method of claim 2 , wherein extracting the detail mask comprises extracting, utilizing the second neural network branch, the detail mask based on the coarse semantic mask, the intermediate image features, and the initial image features extracted from the downsampled image.
4. The method of claim 3 , wherein generating the intermediate image matte comprises combining semantic features of the coarse semantic mask and detail features of the detail mask utilizing the third neural network branch.
5. The method of claim 4 , wherein combining the semantic features of the coarse semantic mask and the detail features of the detail mask comprises combining detail features from the detail mask at one or more boundaries of an object with features corresponding to the object from the coarse semantic mask into the intermediate image matte.
6. The method of claim 1 , further comprising:
comparing, utilizing a semantic discriminator neural network, the coarse semantic mask generated by the first neural network branch to a ground-truth semantic mask corresponding to the digital image;
generating, utilizing the semantic discriminator neural network, a discriminator classification corresponding to the coarse semantic mask relative to the ground-truth semantic mask;
determining a matte adversarial loss based on the discriminator classification; and
learning parameters of the first neural network branch utilizing the matte adversarial loss based on the discriminator classification.
7. The method of claim 1 , further comprising:
comparing, utilizing a matte discriminator neural network, a set of patches of the intermediate image matte to an additional set of patches of a ground-truth image matte;
generating, utilizing the matte discriminator neural network, a plurality of discriminator classifications based on comparisons of the set of patches of the intermediate image matte to the additional set of patches of the ground-truth image matte;
generating a loss function comprising a sum of a plurality of separate loss functions corresponding to the coarse semantic mask, the detail mask, and the intermediate image matte; and
learning parameters of one or more branches of the generative neural network based on the plurality of discriminator classifications and the loss function.
8. The method of claim 1 , further comprising:
determining indications of one or more erroneous regions of the intermediate image matte at one or more edges at a boundary between a foreground region and a background region based on the predicted absolute residual errors from the error map; and
generating, utilizing the refinement neural network, the final image matte by refining one or more patches of the intermediate image matte corresponding to the one or more erroneous regions.
9. The method of claim 8 , wherein generating the final image matte comprises:
determining the one or more patches of the intermediate image matte based on error values in the error map;
refining, utilizing the refinement neural network, the one or more patches of the intermediate image matte without refining one or more additional patches of the intermediate image matte; and
generating the final image matte by upsampling the one or more patches refined utilizing the refinement neural network and the one or more additional patches of the intermediate image matte.
10. A system comprising:
a memory component; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
extracting, utilizing a first neural network branch of a generative neural network, a coarse semantic mask and intermediate image features from a digital image;
extracting, utilizing a second neural network branch of the generative neural network, a detail mask from the coarse semantic mask and the intermediate image features generated by the first neural network branch and initial image features extracted from the digital image;
generating, utilizing a third neural network branch of the generative neural network; an intermediate image matte by fusing semantic features from the coarse semantic mask and detail features from the detail mask; and
generating a final image matte by:
dividing the intermediate image matte into a plurality of patches, having a predetermined size and corresponding to different coordinates of the intermediate image matte;
determining a selected subset of patches of the plurality of patches based on error values corresponding to the plurality of patches from an error map comprising predicted absolute residual errors of pixels of the intermediate image matte; and
refining the selected subset of patches of the intermediate image matte utilizing a refinement neural network.
11. The system of claim 10 , wherein extracting the coarse semantic mask comprises:
generating, utilizing a downsampling neural network layer, a downsampled image at a resolution lower than an original resolution of the digital image; and
extracting, utilizing the first neural network branch, the coarse semantic mask and the intermediate image features from the downsampled image.
12. The system of claim 10 , wherein extracting the detail mask comprises:
determining multi-resolution image features based on the initial image features and the intermediate image features; and
extracting, utilizing the second neural network branch, the detail mask based on the multi-resolution image features and the coarse semantic mask.
13. The system of claim 10 , wherein generating the intermediate image matte comprises:
determining, utilizing the third neural network branch, the semantic features of the coarse semantic mask;
determining, utilizing the third neural network branch, combined features based on the semantic features of the coarse semantic mask and the detail features of the detail mask from the second neural network branch; and
generating, utilizing the third neural network branch, the intermediate image matte based on the combined features.
14. The system of claim 10 , wherein generating the final image matte comprises:
determining the selected subset of patches of the intermediate image matte in response to the selected subset of patches comprising predicted error values above an error threshold; and
combining the refined selected subset of patches of the intermediate image matte with one or more unrefined patches of the intermediate image matte.
15. The system of claim 10 , wherein the operations further comprise:
generating, utilizing a semantic discriminator neural network, a discriminator classification based on a comparison of the coarse semantic mask to a ground-truth semantic mask; and
learning parameters of the first neural network branch based on the discriminator classification.
16. The system of claim 10 , wherein the operations further comprise:
generating, utilizing a matte discriminator neural network, discriminator classifications based on comparisons of a set of patches of the intermediate image matte to patches of a ground-truth image matte; and
learning parameters of the first neural network branch, the second neural network branch, or the third neural network branch of the generative neural network based on the discriminator classifications.
17. A non-transitory computer readable medium comprising instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
extracting, utilizing a generative neural network comprising a plurality of neural network branches, a coarse semantic mask and intermediate image features from a digital image;
extracting, utilizing the generative neural network, a detail mask based on the coarse semantic mask, the intermediate image features, and initial image features extracted from the digital image;
generating, utilizing the generative neural network; an intermediate image matte by fusing semantic features from the coarse semantic mask and detail features from the detail mask; and
generating a final image matte by:
dividing the intermediate image matte into a plurality of patches, having a predetermined size and corresponding to different coordinates of the intermediate image matte;
determining a selected subset of patches of the plurality of patches based on error values corresponding to the plurality of patches from an error map comprising predicted absolute residual errors of pixels of the intermediate image matte; and
refining the selected subset of patches of the intermediate image matte utilizing a refinement neural network.
18. The non-transitory computer readable medium of claim 17 , wherein extracting the coarse semantic mask comprises:
generating, from the digital image, a downsampled image at a resolution lower than a resolution of the digital image; and
extracting, utilizing a semantic neural network branch of the generative neural network, the coarse semantic mask and the intermediate image features from the downsampled image.
19. The non-transitory computer readable medium of claim 18 , wherein extracting the detail mask comprises:
extracting the initial image features from the downsampled image; and
extracting, utilizing a detail neural network branch of the generative neural network, the detail mask based on the coarse semantic mask and multi-resolution image features comprising the initial image features and the intermediate image features.
20. The non-transitory computer readable medium of claim 18 , wherein generating the final image matte comprises:
generating scores for the plurality of patches of the intermediate image matte based on the error values from the error map; and
determining the selected subset of patches of in response to determining that scores of the selected subset of patches exceed an error threshold.