Systems and methods for panoptic segmentation of images for autonomous driving
Systems and methods for generating a panoptic segmentation mask for an input image. The methods include receiving the input image comprising a plurality of pixels, generating a semantic mask and an instance mask from the input image, and combining the semantic mask and the instance mask to generate a panoptic mask for the input image. The semantic mask includes a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels. The instance mask includes a plurality of masks, where each of the plurality of masks identifies an instance of a countable object in the input image, and is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image.
1. A method for panoptic image segmentation, the method comprising:
receiving an input image comprising a plurality of pixels;
generating, from the input image, a semantic mask and an instance mask, wherein:
the semantic mask comprises a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels, and
the instance mask comprises a plurality of masks, wherein each of the plurality of masks:
identifies an instance of a countable object in the input image, and
is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image; and
combining the semantic mask and the instance mask to generate a panoptic mask for the input image.
2. The method of claim 1 , wherein each of the plurality of labels includes a semantic indication of a type of a countable object or amorphous region for each pixel in the input image.
3. The method of claim 1 , further comprising using the panoptic mask to control operations of a mobile platform.
4. The method of claim 1 , wherein the instance mask further comprises a plurality of bounding boxes, each of the plurality of bounding boxes being associated with an instance of one of a plurality of countable objects in the input image, the method further comprising:
determine the instance mask as invalid responsive to an overlap between a first bounding box and a second bounding box is less than a threshold indicative of a first countable object associated with the first bounding box is hidden behind a second countable object associated with the second countable object, wherein the second countable object is transparent.
5. The method of claim 1 , further comprising:
generating the instance mask using an instance segmentation neural network; and
generating the semantic mask with a semantic segmentation neural network.
6. The method of claim 5 , further comprising training the semantic segmentation neural network and the instance segmentation neural network using a ground truth data set comprising a collection of ground truth examples, each ground truth example comprising:
a training image;
a ground truth panoptic mask corresponding to the training image; and
a plurality of instance labels, each of the plurality of instance labels comprising a modal bounding box for one of a plurality of countable objects in the training image, a class label, and pixel id associations with the ground truth panoptic mask.
7. The method of claim 6 , further comprising generating a ground truth semantic mask from the ground truth panoptic mask for training the semantic segmentation neural network.
8. The method of claim 6 , further comprising generating, from the ground truth panoptic mask and the plurality of instance labels, a ground truth instance mask for training the instance segmentation neural network, the ground truth instance mask comprising: a mask for each modal bounding box and an indication of mask validity for that modal bounding box.
9. The method of claim 8 , wherein:
the indication of mask validity for that modal bounding box provides information relating to whether or not that modal bounding box includes an instance of a countable object of the training image that is not hidden behind other objects in the training image; and
the indication of mask validity is determined by computing, for each instance in the ground truth panoptic mask, an overlap between an outline encompassing that instance and a corresponding modal bounding box.
10. The method of claim 9 , further comprising training the instance segmentation neural network by conditioning loss propagation during training upon the indication of mask validity such that no loss is propagated for predicting, by the instance segmentation neural network, a bounding box for an instance of a countable object that is hidden behind other objects in the training image.
11. The method of claim 9 , further comprising determining that a mask is not valid when the overlap is less than a threshold.
12. A system for panoptic image segmentation, the system comprising:
a processor; and
a non-transitory computer readable medium comprising programming instructions that when executed by the processor, will cause the processor to:
receive an input image comprising a plurality of pixels;
generate, from the input image, a semantic mask and an instance mask, wherein:
the semantic mask comprises a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels, and
the instance mask comprises a plurality of masks, wherein each of the plurality of masks:
identifies an instance of a countable object in the input image, and
is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image; and
combine the semantic mask and the instance mask to generate a panoptic mask for the input image,
wherein the instance mask further comprises a plurality of bounding boxes, each of the plurality of bounding boxes being associated with an instance of one of a plurality of countable objects in the input image, the non-transitory computer readable medium further comprising programming instructions that when executed by the processor, will cause the processor to:
determine the instance mask as invalid responsive to an overlap between a first bounding box and a second bounding box is less than a threshold indicative of a first countable object associated with the first bounding box is hidden behind a transparent amorphous region.
13. The system of claim 12 , further comprising programming instructions that when executed by the processor, will cause the processor to use the panoptic mask to control operations of a mobile platform.
14. The system of claim 12 , further comprising programming instructions that when executed by the processor, will cause the processor to:
generate the instance mask using an instance segmentation neural network; and
generate the semantic mask with a semantic segmentation neural network.
15. The system of claim 14 , further comprising programming instructions that when executed by the processor, will cause the processor to train the semantic segmentation neural network and the instance segmentation neural network using a ground truth data set comprising a collection of ground truth examples, each ground truth example comprising:
a training image;
a ground truth panoptic mask corresponding to the training image; and
a plurality of instance labels, each of the plurality of instance labels comprising a modal bounding box for one of a plurality of countable objects in the training image, a class label, and pixel id associations with the ground truth panoptic mask.
16. The system of claim 15 , further comprising programming instructions that when executed by the processor, will cause the processor to generate a ground truth semantic mask from the ground truth panoptic mask for training the semantic segmentation neural network.
17. The system of claim 15 , further comprising programming instructions that when executed by the processor, will cause the processor to generate, from the ground truth panoptic mask and the plurality of instance labels, a ground truth instance mask for training the instance segmentation neural network, the ground truth instance mask comprising: a mask for each modal bounding box and an indication of mask validity for that modal bounding box.
18. The system of claim 17 , wherein:
the indication of mask validity for that modal bounding box provides information relating to whether or not that modal bounding box includes an instance of a countable object of the training image that is not hidden behind other objects in the training image; and
the indication of mask validity is determined by computing, for each instance in the ground truth panoptic mask, an overlap between an outline encompassing that instance and a corresponding modal bounding box.
19. The system of claim 18 , further comprising programming instructions that when executed by the processor, will cause the processor to train the instance segmentation neural network by conditioning loss propagation during training upon the indication of mask validity such that no loss is propagated for predicting, by the instance segmentation neural network, a bounding box for an instance of a countable object that is hidden behind other objects in the training image.
20. A computer program product comprising a memory and programming instructions that are configured to cause a processor to:
receive an input image comprising a plurality of pixels;
generate, from the input image, a semantic mask and an instance mask, wherein:
the semantic mask comprises a single-channel mask that associates each pixel in the input image with a corresponding one of a plurality of labels, and
the instance mask comprises a plurality of masks, wherein each of the plurality of masks:
identifies an instance of a countable object in the input image, and
is associated with an indication of whether that instance of the countable object is hidden behind another object in the input image; and
combine the semantic mask and the instance mask to generate a panoptic mask for the input image.