Infrastructure analysis using panoptic segmentation
Methods and systems identifying road hazards include capturing an image of a road scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. A mask is generated for an object within the image using the segmentation model. A probability is determined that the object matches a road hazard using the segmentation mode. A signal is generated responsive to the probability to ameliorate a danger posed by the road hazard.
1 . A computer-implemented method for identifying road hazards, comprising:
capturing an image of a road scene using a camera;
embedding the image using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space;
generating a mask for an object within the image using the segmentation model;
determining a probability that the object matches a road hazard using the segmentation mode; and
generating a signal responsive to the probability to ameliorate a danger posed by the road hazard.
2 . The method of claim 1 , wherein the joint latent space represents a visual object and a textual description of visual objects as vectors that are similar to one another according to a distance metric.
3 . The method of claim 2 , wherein determining the probability includes comparing masks to known road hazards using a distance metric in the joint latent space.
4 . The method of claim 3 , wherein the known road hazards are represented in the joint latent space as embeddings of textual descriptions of the road hazards.
5 . The method of claim 1 , wherein the distance metric is a cosine distance.
6 . The method of claim 1 , wherein generating the mask includes identifying a location of the object in the image and generating a label for the object.
7 . The method of claim 1 , wherein generating the signal includes transmitting the signal to a system selected from the group consisting of a vehicle, roadside signage, and a municipal authority.
8 . The method of claim 1 , wherein generating the signal includes determining that the probability exceeds a threshold value to determine that the object shows a road hazard.
9 . The method of claim 1 , further comprising ameliorating the road hazard by performing an action selected from the group consisting of displaying a warning on roadside signage, audibly announcing the road hazard, and deploying a cleanup crew to correct the road hazard.
10 . The method of claim 1 , wherein the segmentation model is trained on a plurality of training datasets that include a panoptic segmentation dataset, which includes class labels for individual image pixels, and an object detection dataset, which includes a class label for a bounding box.
11 . A system for identifying road hazards, comprising:
a hardware processor; and
a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
capture an image of a road scene using a camera;
embed the image using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space;
generate a mask for an object within the image using the segmentation model;
determine a probability that the object matches a road hazard using the segmentation mode; and
generate a signal responsive to the probability to ameliorate a danger posed by the road hazard.
12 . The system of claim 11 , wherein the joint latent space represents a visual object and a textual description of visual objects as vectors that are similar to one another according to a distance metric.
13 . The system of claim 12 , wherein the computer program further causes the hardware processor to determine compare masks to known road hazards using a distance metric in the joint latent space.
14 . The system of claim 13 , wherein the known road hazards are represented in the joint latent space as embeddings of textual descriptions of the road hazards.
15 . The system of claim 11 , wherein the distance metric is a cosine distance.
16 . The system of claim 11 , wherein the computer program further causes the hardware processor to identify a location of the object in the image and to generate a label for the object.
17 . The system of claim 11 , wherein the computer program further causes the hardware processor to transmit the signal to a system selected from the group consisting of a vehicle, roadside signage, and a municipal authority.
18 . The system of claim 11 , wherein the computer program further causes the hardware processor to determine that the probability exceeds a threshold value to determine that the object shows a road hazard.
19 . The system of claim 11 , wherein the computer program further causes the hardware processor to ameliorate the road hazard by performing an action selected from the group consisting of displaying a warning on roadside signage, audibly announcing the road hazard, and deploying a cleanup crew to correct the road hazard.
20 . The system of claim 11 , wherein the segmentation model is trained on a plurality of training datasets that include a panoptic segmentation dataset, which includes class labels for individual image pixels, and an object detection dataset, which includes a class label for a bounding box.