Fast one-shot open vocabulary image-conditioned detection and search method for utility assets
This disclosure describes a system, method, and non-transitory computer-readable medium for image search-based object detection of utility assets in image databases. The method includes receiving an input image of a utility asset and a query bounding box representing an image-based object query. Bounding boxes of objects represented in the input image are generated based on the input image and the query bounding box, in which anchoring boxes corresponding to object classifications are identified from the bounding boxes. A textual label is determined for a selected subset of anchoring boxes. An image embedding representing the region is encoded, and image tokens are generated based on the encoded image embedding. Output images of other utility assets relevant to the image-based object query are identified from images in an image database, based on at least one of (i) the encoded image embedding, (ii) the image tokens, or (iii) the textual label. The output images are provided for output.
1 . A method for image search-based object detection, the method comprising:
receiving input data comprising an input image of a utility asset and a query bounding box, the query bounding box enclosing a region of the input image that contains a portion of the utility asset representing an image-based object query;
generating bounding boxes based on the input image and the query bounding box, the bounding boxes comprising one or more objects represented in the input image;
identifying anchoring boxes based on the bounding boxes, the anchoring boxes corresponding to classifications of the one or more objects;
selecting, based on the query bounding box, a subset of anchoring boxes from the anchoring boxes;
generating an encoded query image embedding of the region of the input image represented by the query bounding box, the encoded query image embedding representing the region of the input image;
generating one or more image tokens from the encoded query image embedding of the query bounding box, each of the one or more image tokens representing an input element of the query bounding box;
filtering, from an image database comprising a plurality of images and based on the one or more image tokens, the plurality of images to obtain a subset of images, wherein the subset of images excludes images that do not include at least one token corresponding with the one or more image tokens; and
determining, from the subset of images and based on the encoded query image embedding, one or more output images of other utility assets relevant to the image-based object query, wherein the one or more output images comprise an image embedding that at least partially matches the encoded query image embedding; and
providing, as output, the one or more output images.
2 . The method of claim 1 , further comprising:
searching, based on a textual label representing the subset of anchoring boxes, for a set of images from the plurality of images in the image database, wherein the set of images excludes one or more images from the pluarlity of images in the image database that do not include a label that matches the textual label;
determining, based on the encoded query image embedding, a similarity score for each image in the set of images, wherein the similarity score indicates a likelihood of a respective image having an embedding matching the encoded query image embedding of the input image; and
identifying the one or more output images from the set of images, the one or more output images each having a respective similarity score exceeds a threshold value.
3 . The method of claim 2 , wherein identifying the one or more output images comprises performing a nearest neighbor search of one or more image embeddings for the set of images from the encoded query image embedding of the query bounding box.
4 . The method of claim 2 , wherein further comprising ranking the one or more output images based on the respective similarity score of each output image in the one or more output images.
5 . The method of claim 2 , wherein the textual label for the subset of anchoring boxes is determined from annotations representing one or more detected objects in the subset of anchoring boxes.
6 . The method of claim 2 , wherein determining the textual label for the subset of anchoring boxes comprises:
comparing, by a neural network configured to generate textual labels, a ground truth label for the subset of anchoring boxes to the textual label to obtain a comparison result; and
updating one or more parameters of the neural network based on the comparison result.
7 . The method of claim 1 , wherein the utility asset is at least one of (i) a utility pole, (ii) a transformer, (iii) one or more wires, or (iv) other types of electrical grid distribution equipment.
8 . The method of claim 1 , wherein the input image of the utility asset captures at least one of (i) a type of defect, or (ii) one or more instances of a defect, of the utility asset.
9 . The method of claim 1 , wherein the query bounding box enclosing the region of the input image is provided by at least one of (i) a user interface, or (ii) a set of coordinates, the user interface configured to provide the input image for display and the set of coordinates representing locations of pixels corresponding to the region of the input image enclosed by the query bounding box.
10 . The method of claim 1 , wherein generating the bounding boxes based on the input image and the query bounding box comprises providing a training example to a model configured to perform object detection, wherein the training example comprises (i) a label for a respective classification of the one or more objects indicating identification of a utility asset and (ii) a sparse set of image data representing the utility asset.
11 . The method of claim 1 , wherein selecting the subset of anchoring boxes from the anchoring boxes comprises:
determining that the subset of anchoring boxes match the query bounding box within a threshold value; and
based on determining that the subset of anchoring boxes do not match the query bounding box within the threshold value, updating the anchoring boxes by a model configured to perform auto-anchoring of anchoring boxes.
12 . The method of claim 1 , wherein generating the encoded query image embedding of the query bounding box comprises:
generating, by a visual transformer encoder and based on the encoded query image embedding for the bounding boxes, a reconstructed image;
comparing the reconstructed image to the input image to obtain a comparison result; and
updating one or more parameters of the visual transformer encoder based on the comparison result.
13 . A system for performing object detection and search of utility assets in an image database, the system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving input data comprising an input image of a utility asset and a query bounding box, the query bounding box enclosing a region of the input image that contains a portion of the utility asset representing an image-based object query;
generating bounding boxes based on the input image and the query bounding box, the bounding boxes comprising one or more objects represented in the input image;
identifying anchoring boxes based on the bounding boxes, the anchoring boxes corresponding to classifications of the one or more objects;
selecting, based on the query bounding box, a subset of anchoring boxes from the anchoring boxes;
generating an encoded query image embedding of the region of the input image represented by the query bounding box, the encoded query image embedding representing the region of the input image;
generating one or more image tokens from the encoded query image embedding of the query bounding box, each of the one or more image tokens representing an input element of the query bounding box;
filtering, from an image database comprising a plurality of images and based on the one or more image tokens, the plurality of images to obtain a subset of images, wherein the subset of images excludes images that do not include at least one token corresponding with the one or more image tokens; and
determining, from the subset of images and based on the encoded query image embedding, one or more output images of other utility assets relevant to the image-based object query, wherein the one or more output images comprise an image embedding that at least partially matches the encoded query image embedding; and
providing, as output, the one or more output images.
14 . The system of claim 13 , the operations further comprising:
searching, based on a textual label representing the subset of anchoring boxes, for a set of images from the plurality of images in the image database, wherein the subset-set of images excludes one or more images from the plurality of images in the image database that do not include a label that matches the textual label;
determining, based on the encoded query image embedding, a similarity score for each image in the set of images, wherein the similarity score indicates a likelihood of a respective image having an embedding matching the encoded query image embedding of the input image; and
identifying the one or more output images from the filtered subset-set of images, the one or more output images each having a respective similarity score exceeds a threshold value.
15 . The system of claim 14 , wherein identifying the one or more output images comprises performing a nearest neighbor search of one or more image embeddings for the set of images from the encoded query image embedding of the query bounding box.
16 . The system of claim 14 , wherein providing the one or more output images further comprises:
ranking the one or more output images based on the respective similarity score of each output image in the one or more output images.
17 . The system of claim 14 , wherein the textual label for the subset of anchoring boxes is determined from annotations representing one or more detected objects in the subset of anchoring boxes.
18 . The system of claim 13 , wherein the query bounding box enclosing the region of the input image is provided by a user interface.
19 . A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:
receiving input data comprising an input image of a utility asset and a query bounding box, the query bounding box enclosing a region of the input image that contains a portion of the utility asset representing an image-based object query;
generating bounding boxes based on the input image and the query bounding box, the bounding boxes comprising one or more objects represented in the input image;
identifying anchoring boxes based on the bounding boxes, the anchoring boxes corresponding to classifications of the one or more objects;
selecting, based on the query bounding box, a subset of anchoring boxes from the anchoring boxes;
generating an encoded query image embedding of the region of the input image represented by the query bounding box, the encoded query image embedding representing the region of the input image;
generating one or more image tokens from the encoded query image embedding of the query bounding box, each of the one or more image tokens representing an input element of the query bounding box;
filtering, from an image database comprising a plurality of images and based on the one or more image tokens, the plurality of images to obtain a subset of images, wherein the subset of images excludes images that do not include at least one token corresponding with the one or more image tokens; and
determining, from the subset of images and based on the encoded query image embedding, one or more output images of other utility assets relevant to the image-based object query, wherein the one or more output images comprise an image embedding that at least partially matches the encoded query image embedding; and
providing, as output, the one or more output images.
20 . The non-transitory, computer-readable medium of claim 19 , the operations further comprising:
searching, based on a textual label representing the subset of anchoring boxes, for a set of images from the plurality of images in the image database, wherein the set of images excludes one or more images from the plurality of images in the image database that do not include a label that matches the textual label;
determining, based on the encoded query image embedding, a similarity score for each image in the set of images, wherein the similarity score indicates a likelihood of a respective image having an embedding matching the encoded query image embedding of the input image; and
identifying the one or more output images from the set of images, the one or more output images each having a respective similarity score exceeds a threshold value.