Utilizing implicit neural representations to parse visual components of subjects depicted within visual content
This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that utilize a local implicit image function neural network to perform image segmentation with a continuous class label probability distribution. For example, the disclosed systems utilize a local-implicit-image-function (LIIF) network to learn a mapping from an image to its semantic label space. In some instances, the disclosed systems utilize an image encoder to generate an image vector representation from an image. Subsequently, in one or more implementations, the disclosed systems utilize the image vector representation with a LIIF network decoder that generates a continuous probability distribution in a label space for the image to create a semantic segmentation mask for the image. Moreover, in some embodiments, the disclosed systems utilize the LIIF-based segmentation network to generate segmentation masks at different resolutions without changes in an input resolution of the segmentation network.
1. A computer-implemented method comprising:
generating, utilizing an image encoder, an image vector representation from an image depicting a subject;
utilizing a local implicit image function neural network to generate a continuous class label probability distribution for one or more class labels from the image vector representation; and
creating a semantic segmentation mask for the image comprising one or more labeled semantic regions based on the continuous class label probability distribution.
2. The computer-implemented method of claim 1 , further comprising utilizing the continuous class label probability distribution for the one or more class labels to determine a class prediction for a coordinate between pixels of the image.
3. The computer-implemented method of claim 1 , further comprising:
generating an unfolded image vector representation from the image vector representation utilizing an unfolding operation;
generating a reduced channel image vector representation by utilizing one or more multilayer perceptron decoders to reduce channels of the unfolded image vector representation; and
generating the continuous class label probability distribution for the one or more class labels from the reduced channel image vector representation.
4. The computer-implemented method of claim 1 , further comprising:
generating a global pool feature vector from the image vector representation utilizing global pooling; and
generating the continuous class label probability distribution for the one or more class labels from the global pool feature vector.
5. The computer-implemented method of claim 1 , wherein the image encoder comprises:
a residual block comprising instance normalization layers and convolution layers; and
strided convolution layers between one or more residual blocks.
6. The computer-implemented method of claim 1 , further comprising:
selecting a plurality of upsample coordinates for generating the semantic segmentation mask at an upsampled resolution; and
creating the semantic segmentation mask by generating semantic label predictions at the plurality of upsample coordinates utilizing the continuous class label probability distribution.
7. The computer-implemented method of claim 1 , further comprising utilizing the semantic segmentation mask to edit the one or more labeled semantic regions in the image.
8. The computer-implemented method of claim 1 , further comprising learning parameters of the local implicit image function neural network utilizing an edge-aware loss using a ground truth image with edges for known semantic regions of the ground truth image.
9. A system comprising:
a memory component comprising an image depicting a subject, a convolutional image encoder, and a local implicit image function neural network; and
a processing device coupled to the memory component, the processing device to perform operations comprising:
generating, utilizing the convolutional image encoder, an image vector representation from the image depicting the subject;
generating, utilizing the local implicit image function neural network, a continuous class label probability distribution for one or more class labels from the image vector representation;
selecting a plurality of upsample coordinates for generating a semantic segmentation mask at an upsampled resolution; and
creating a semantic segmentation mask by generating semantic label predictions at the plurality of upsample coordinates utilizing the continuous class label probability distribution.
10. The system of claim 9 , wherein the operations further comprise generating the image depicting the subject by downsampling a higher resolution image depicting the subject.
11. The system of claim 9 , wherein the operations further comprise utilizing the continuous class label probability distribution for the one or more class labels to determine a class prediction for an upsample coordinate from the plurality of upsample coordinates between pixels of the image.
12. The system of claim 9 , wherein the operations further comprise generating, utilizing the local implicit image function neural network, the continuous class label probability distribution by utilizing a reduced channel image vector representation based on the image vector representation and a global pool feature vector based on the image vector representation.
13. The system of claim 9 , wherein the convolutional image encoder comprises a residual block comprising instance normalization layers and convolution layers.
14. The system of claim 9 , wherein the semantic segmentation mask comprises one or more labeled semantic regions based on the semantic label predictions at the plurality of upsample coordinates and wherein the operations further comprise utilizing the semantic segmentation mask to edit the one or more labeled semantic regions in a higher resolution image of the image.
15. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
generating, utilizing an image encoder, an image vector representation from an image depicting a human face;
generating, utilizing a local implicit image function neural network, a continuous class label probability distribution for one or more facial feature labels; and
creating a semantic segmentation mask for the image comprising one or more labeled facial feature regions based on the continuous class label probability distribution.
16. The non-transitory computer-readable medium of claim 15 , wherein the one or more facial feature labels comprise an eye label, a nose label, a lips label, a skin label, an eyebrows label, a teeth label, and a hair label.
17. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise utilizing the continuous class label probability distribution for the one or more facial feature labels to determine a facial feature prediction for a coordinate between pixels of the image.
18. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise generating, utilizing the local implicit image function neural network, the continuous class label probability distribution based on:
a reduced channel image vector representation generated utilizing one or more multilayer perceptron decoders with the image vector representation; and
a global pool feature vector generated utilizing global pooling on the image vector representation.
19. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise creating the semantic segmentation mask at an upsampled resolution by generating semantic label predictions at a plurality of upsample coordinates utilizing the continuous class label probability distribution.
20. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise utilizing the semantic segmentation mask to edit the one or more labeled facial feature regions in the image.