Fusing predictions for end-to-end panoptic segmentation
A method for controlling a vehicle based on a panoptic map includes receiving an input from at least one sensor of the vehicle. The method also includes generating an instance map and a semantic map from the input. The method further includes generating the panoptic map from the instance map and the semantic map based on a binary mask. The method still further includes controlling the vehicle based on the panoptic map.
1. A method for controlling a vehicle based on a panoptic map, comprising:
receiving an input from at least one sensor of the vehicle;
generating an instance map and a semantic map from the input;
generating, based on the input, a context map identifying at least one of scene depth, an edge of the objects, surface normals of the objects, or an optical flow of the objects;
generating a binary mask based on the input, the instance map, and the semantic map;
generating the panoptic map by applying the binary mask to the instance map, the context map, and the semantic map; and
controlling the vehicle based on the panoptic map.
2. The method of claim 1 , in which:
the instance map identifies each instance of a countable object; and
the semantic map associates each pixel in the input with one of a plurality of labels.
3. The method of claim 1 , further comprising generating the instance map and the semantic map with a different neural network.
4. The method of claim 1 , further comprising generating the binary mask with an artificial neural network.
5. The method of claim 4 , in which the binary mask is used to determine whether a pixel is associated with a uniquely identifiable instance of an object in the input.
6. The method of claim 4 , further comprising training the artificial neural network to generate the binary mask based on a training input labeled with object instances.
7. An apparatus for controlling a vehicle based on a panoptic map, the apparatus comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor configured:
to receive an input from at least one sensor of the vehicle;
to generate an instance map and a semantic map from the input;
to generate, based on the input, a context map identifying at least one of scene depth, an edge of the objects, surface normals of the objects, or an optical flow of the objects;
to generate a binary mask based on the input, the instance map, and the semantic map;
to generate the panoptic map by applying the binary mask to the instance map, the context map, and the semantic map; and
to control the vehicle based on the panoptic map.
8. The apparatus of claim 7 , in which:
the instance map identifies each instance of a countable object; and
the semantic map associates each pixel in the input with one of a plurality of labels.
9. The apparatus of claim 7 , in which the at least one processor is further configured to generate the instance map and the semantic map with a different neural network.
10. The apparatus of claim 7 , in which the at least one processor is further configured to generate the binary mask with an artificial neural network.
11. The apparatus of claim 10 , in which the binary mask is used to determine whether a pixel is associated with a uniquely identifiable instance of an object in the input.
12. The apparatus of claim 10 , in which the at least one processor is further configured to train the artificial neural network to generate the binary mask based on a training input labeled with object instances.
13. A non-transitory computer-readable medium having program code recorded thereon for controlling a vehicle based on a panoptic map, the program code executed by a processor and comprising:
program code to receive an input from at least one sensor of the vehicle;
program code to generate an instance map and a semantic map from the input;
program code to generate, based on the input, a context map identifying at least one of scene depth, an edge of the objects, surface normals of the objects, or an optical flow of the objects;
program code to generate a binary mask based on the input, the instance map, and the semantic map;
program code to generate the panoptic map by applying the binary mask to the instance map, the context map, and the semantic map; and
program code to control the vehicle based on the panoptic map.
14. The non-transitory computer-readable medium of claim 13 , in which:
the instance map identifies each instance of a countable object; and
the semantic map associates each pixel in the input with one of a plurality of labels.
15. The non-transitory computer-readable medium of claim 13 , in which the program code further comprises program code to generate the binary mask with an artificial neural network.