Semantic image capture fault detection
Methods and systems for detecting faults include capturing an image of a scene using a camera. The image is embedded using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space. Semantic information is generated for a region of the image corresponding to a predetermined static object using the embedded image. A fault of the camera is identified based on a discrepancy between the semantic information and semantic information of the predetermined static image. The fault of the camera is corrected.
1. A computer-implemented method for detecting faults, comprising:
capturing an image of a scene using a camera;
embedding the image using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space;
generating semantic information for a region of the image corresponding to a predetermined static object using the embedded image;
identifying a fault of the camera based on a discrepancy between the semantic information and semantic information of the predetermined static image; and
correcting the fault of the camera.
2. The method of claim 1 , wherein the joint latent space represents a visual object and a textual description of visual objects as vectors that are similar to one another according to a distance metric.
3. The method of claim 2 , wherein the semantic information for the region and the semantic information of the predetermined static image each includes a textual label that is similar to the region and predetermined static object in the joint latent space.
4. The method of claim 3 , wherein identifying the fault includes determining that the textual label of the static image differs from the textual label of the region.
5. The method of claim 3 , wherein the semantic information for the region and the semantic information of the predetermined static object each includes a respective confidence score for the corresponding textual label.
6. The method of claim 5 , wherein identifying the fault includes determining that the confidence score for the textual label of the predetermined static object differs from the confidence score for the textual label of the region differs by more than a threshold amount.
7. The method of claim 2 , wherein the distance metric is a cosine distance.
8. The method of claim 1 , further comprising identifying the predetermined static object within a prior image, taken at an earlier point in time than the captured image.
9. The method of claim 1 , wherein the segmentation model is trained on a plurality of training datasets that include a panoptic segmentation dataset, which includes class labels for individual image pixels, and an object detection dataset, which includes a class label for a bounding box.
10. The method of claim 1 , wherein correcting the fault comprises performing an automatic action selected from the group consisting of performing a self-diagnostic within the camera and activating a self-cleaning function.
11. A system for detecting faults, comprising:
a hardware processor; and
a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
capture an image of a scene using a camera;
embed the image using a segmentation model that includes an image branch having an image embedding layer that embeds images into a joint latent space and a text branch having a text embedding layer that embeds text into the joint latent space;
generate semantic information for a region of the image corresponding to a predetermined static object using the embedded image;
identify a fault of the camera based on a discrepancy between the semantic information and semantic information of the predetermined static image; and
correct the fault of the camera.
12. The system of claim 11 , wherein the joint latent space represents a visual object and a textual description of visual objects as vectors that are similar to one another according to a distance metric.
13. The system of claim 12 , wherein the semantic information for the region and the semantic information of the predetermined static image each includes a textual label that is similar to the region and predetermined static object in the joint latent space.
14. The system of claim 13 , wherein the computer program further causes the hardware processor to determine that the textual label of the static image differs from the textual label of the region.
15. The system of claim 13 , wherein the semantic information for the region and the semantic information of the predetermined static object each includes a respective confidence score for the corresponding textual label.
16. The system of claim 15 , wherein the computer program further causes the hardware processor to determine that the confidence score for the textual label of the predetermined static object differs from the confidence score for the textual label of the region differs by more than a threshold amount.
17. The system of claim 12 , wherein the distance metric is a cosine distance.
18. The system of claim 11 , wherein the computer program further causes the hardware processor to identify the predetermined static object within a prior image, taken at an earlier point in time than the captured image.
19. The system of claim 11 , wherein the segmentation model is trained on a plurality of training datasets that include a panoptic segmentation dataset, which includes class labels for individual image pixels, and an object detection dataset, which includes a class label for a bounding box.
20. The system of claim 11 , wherein the computer program further causes the hardware processor to perform an automatic action selected from the group consisting of performing a self-diagnostic within the camera and activating a self-cleaning function.