Enhanced semantic segmentation of images

1. A non-transitory computer-readable storage medium having instructions stored thereon for semantic segmentation, which, when executed by a processor of a computing device cause the computing device to perform actions comprising:

receiving image data comprising a set of pixels, wherein a first subset of the set of pixels is associated with an object;

generating a coarse segmentation mask of a first resolution by identifying a second subset of the set of pixels approximating the first subset of the set of pixels;

using a segmentation refinement model to generate an intermediate segmentation mask of a second resolution based on the image data and the coarse segmentation mask;

using the segmentation refinement model to generate a refined segmentation mask of the first resolution based on the intermediate segmentation mask, the image data, and the coarse segmentation mask; and

identifying a third subset of the set of pixels using the refined segmentation mask, wherein the third subset of the set of pixels is a closer approximation of the first subset of the set of pixels than the second subset of the set of pixels.

2. The computer-readable storage medium of claim 1 , wherein using the segmentation refinement model to iteratively generate the refined segmentation mask comprises:

generating downsized image data that encodes a downsized image of the image data, wherein the downsized image data is of the second resolution that is less than the first resolution of the image data;

generating a downsized segmentation mask by downsizing the coarse segmentation mask, wherein the downsized segmentation mask is of the second resolution; and

concatenating the downsized image data and the downsized segmentation mask.

3. The computer-readable storage medium of claim 1 , wherein generating the refined segmentation mask based on the intermediate segmentation mask comprises:

generating midsized image data that encodes a midsized image of the image data, wherein the midsized image is of a third resolution that is less than the first resolution of the image data and greater than the second resolution;

generating a midsized segmentation mask via upsizing the intermediate segmentation mask, wherein the midsized segmentation mask is of the third resolution;

generating a concatenation of the midsized image data and the midsized segmentation mask;

using the segmentation refinement model to generate a second intermediate segmentation mask based on the concatenation of the midsized image data and the midsized segmentation mask, wherein the second intermediate segmentation mask is of the third resolution; and

generating the refined segmentation mask based on an upsizing operation on the second intermediate segmentation mask.

4. The computer-readable storage medium of claim 1 , wherein the segmentation refinement model is implemented by a convolutional encoder-decoder neural network.

5. The computer-readable storage medium of claim 1 , wherein the actions further include:

accessing a training image and a corresponding ground-truth (GT) segmentation mask;

using a coarse segmentation model to generate a training coarse segmentation mask for the training image;

generating a difference map based on a comparison of the training coarse segmentation mask and the GT segmentation mask, wherein the difference map indicates one or more differences between the training coarse segmentation mask and the GT segmentation mask; and

using the difference map, the training coarse segmentation mask, the training image, and the GT segmentation mask to update the segmentation refinement model.

6. The computer-readable storage medium of claim 5 , wherein using the difference map, the training coarse segmentation mask, the training image, and the GT segmentation mask to update the segmentation refinement model comprises:

generating a sampled patch of the training image, a corresponding sampled patch of the training coarse segmentation mask, and a corresponding sampled patch of the GT segmentation mask, wherein each of the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask are associated with a region of the difference map that indicates a first difference of the one or more differences between the training coarse segmentation mask and the GT segmentation mask, wherein the first difference is greater than a predefined difference threshold; and

using the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to update the segmentation refinement model.

7. The computer-readable storage medium of claim 6 , wherein using the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to train the segmentation refinement model comprises:

resizing each of the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to a predefined size;

using the segmentation refinement model to generate a training refined segmentation mask, based on a concatenation of the resized sampled patch of the training image and the resized corresponding sampled patch of the training coarse segmentation mask;

determining a value of a loss function based on a comparison of the training refined segmentation mask and the resized corresponding sampled patch of the GT segmentation mask; and

updating the segmentation refinement model based on the value of the loss function.

8. A method for segmenting an object that is visually depicted via a first subset of a set of pixels encoded by source image data, the method comprising:

steps for receiving a coarse segmentation mask identifying a second subset of the set of pixels, wherein the second subset of the set of pixels is an approximation of the first subset of the set of pixels, and wherein the source image data and the coarse segmentation mask are of a first resolution;

steps for using a segmentation refinement model to generate an intermediate segmentation mask of a second resolution based on the source image data and the coarse segmentation mask;

steps for generating a refined segmentation mask of the first resolution based on the intermediate segmentation mask, the segmentation refinement model, the source image data, and the coarse segmentation mask; and

steps for identifying a third subset of the set of pixels using the refined segmentation mask, wherein the third subset of the set of pixels is a closer approximation of the first subset of the set of pixels than the approximation of the second subset of the set of pixels.

9. The method of claim 8 , wherein the steps for generating the refined segmentation mask comprise:

steps for generating downsized image data that encodes a downsized image of the source image data; and

steps for generating a downsized segmentation mask by downsizing the coarse segmentation mask.

10. The method of claim 9 , wherein the steps for generating the refined segmentation mask comprise:

steps for generating midsized image data that encodes a midsized image of the source image data, wherein the midsized image is of a third resolution that is less than the first resolution of the source image data and greater than the second resolution of the downsized image;

steps for generating a midsized segmentation mask via an upsizing operation on the intermediate segmentation mask, wherein the midsized segmentation mask is of the third resolution;

steps for generating a concatenation of the midsized image data and the midsized segmentation mask;

steps for using the segmentation refinement model to generate a second intermediate segmentation mask based on the concatenation of the midsized image data and the midsized segmentation mask, wherein the second intermediate segmentation mask is of the third resolution; and

steps generating the refined segmentation mask based on an upsizing operation on the second intermediate segmentation mask.

11. The method of claim 8 , further comprising:

steps for accessing a training image and a corresponding ground-truth (GT) segmentation mask;

steps for using a coarse segmentation model to generate a training coarse segmentation mask for the training image;

steps for generating a difference map based on a comparison of the training coarse segmentation mask and the GT segmentation mask, wherein the difference map indicates one or more differences between the training coarse segmentation mask and the GT segmentation mask; and

steps for using the difference map, the training coarse segmentation mask, the training image, and the GT segmentation mask to train the segmentation refinement model.

12. The method of claim 11 , wherein steps for using the difference map, the training coarse segmentation mask, the training image, and the GT segmentation mask to train the segmentation refinement model comprise:

steps for generating a sampled patch of the training image, a corresponding sampled patch of the training coarse segmentation mask, and a corresponding sampled patch of the GT segmentation mask, wherein each of the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask are associated with a region of the difference map that indicates a first difference of the one or more differences between the training coarse segmentation mask and the GT segmentation mask, wherein the first difference is greater than a predefined difference threshold; and

steps for using the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to train the segmentation refinement model.

13. The method of claim 12 , wherein steps for using the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to train the segmentation refinement model comprises:

steps for resizing each of the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to a predefined size;

steps for using the segmentation refinement model to generate a training refined segmentation mask, based on a concatenation of the resized sampled patch of the training image and the resized corresponding sampled patch of the GT segmentation mask;

steps for determining a loss function based on a comparison of the training refined segmentation mask and the resized corresponding sampled patch of the training coarse segmentation mask; and

steps for updating the segmentation refinement model based on the loss function.

14. A computing system for semantic segmentation, comprising:

a processor device; and

a computer-readable storage medium, coupled with the processor device, having instructions stored thereon, which, when executed by the processor device, perform actions comprising:

generating a difference map based on a training coarse segmentation mask and a ground-truth (GT) segmentation mask corresponding to a training image, wherein the difference map indicates one or more differences between the training coarse segmentation mask and the GT segmentation mask;

using the difference map, the training coarse segmentation mask, the training image, and the GT segmentation mask to train a segmentation refinement model;

receiving image data comprising a set of pixels, wherein a first subset of the set of pixels associated with an object;

generating a coarse segmentation mask that segments the image data by identifying a second subset of the set of pixels, wherein the second subset of the set of pixels is an approximation of the first subset of the set of pixels; and

using the segmentation refinement model to generate a refined segmentation mask based on the image data and the coarse segmentation mask.

15. The computing system of claim 14 , wherein the image data and the coarse segmentation mask are of a first resolution and using the segmentation refinement model to iteratively generate the refined segmentation mask comprises:

generating downsized image data that encodes a downsized image of the image data, wherein the downsized image is of a second resolution that is less than the first resolution of the image data;

generating a downsized segmentation mask by downsizing the coarse segmentation mask, wherein the downsized segmentation mask is of the second resolution;

generating a concatenation of the downsized image data and the downsized segmentation mask;

using the segmentation refinement model to generate a first intermediate segmentation mask based on the concatenation of the downsized image data and the downsized segmentation mask, wherein the first intermediate segmentation mask is of the second resolution; and

generating the refined segmentation mask based on an upsizing operation on the first intermediate segmentation mask, wherein the refined segmentation mask is of the first resolution.

16. The computing system of claim 15 , wherein generating the refined segmentation mask based on the upsizing operation on the first intermediate segmentation mask comprises:

generating midsized image data that encodes a midsized image of the image data, wherein the midsized image is of a third resolution that is less than the first resolution of the image data and greater than the second resolution of the downsized image;

generating a midsized segmentation mask via the upsizing operation on the first intermediate segmentation mask, wherein the midsized segmentation mask is of the third resolution;

generating a concatenation of the midsized image data and the midsized segmentation mask;

using the segmentation refinement model to generate a second intermediate segmentation mask based on the concatenation of the midsized image data and the midsized segmentation mask, wherein the second intermediate segmentation mask is of the third resolution; and

generating the refined segmentation mask based on an upsizing operation on the second intermediate segmentation mask.

17. The computing system of claim 14 , wherein the segmentation refinement model is implemented by a convolutional encoder-decoder neural network.

18. The computing system of claim 14 , wherein the actions further include using a coarse segmentation model to generate the training coarse segmentation mask for the training image.

19. The computing system of claim 14 , wherein using the difference map, the training coarse segmentation mask, the training image, and the GT segmentation mask to train the segmentation refinement model comprises:

generating a sampled patch of the training image, a corresponding sampled patch of the training coarse segmentation mask, and a corresponding sampled patch of the GT segmentation mask, wherein each of the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask are associated with a region of the difference map that indicates a first difference of the one or more differences between the training coarse segmentation mask and the GT segmentation mask, wherein the first difference is greater than a predefined difference threshold; and

using the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to train the segmentation refinement model.

20. The computing system of claim 19 , wherein using the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to train the segmentation refinement model comprises:

resizing each of the sampled patch of the training image, the corresponding sampled patch of the training coarse segmentation mask, and the corresponding sampled patch of the GT segmentation mask to a predefined size;

using the segmentation refinement model to generate a training refined segmentation mask, based on a concatenation of the resized sampled patch of the training image and the resized corresponding sampled patch of the training coarse segmentation mask;

determining a loss function based on a comparison of the training refined segmentation mask and the resized corresponding sampled patch of the GT segmentation mask; and updating the segmentation refinement model based on the loss function.