Systems and methods for open vocabulary object detection
Embodiments described herein provide methods and systems for open vocabulary object detection of images. given a pre-trained vision-language model and an image-caption pair, an activation map may be computed in the image that corresponds to an object of interest mentioned in the caption. The activation map is then converted into a pseudo bounding-box label for the corresponding object category. The open vocabulary detector is then directly supervised by these pseudo box-labels, which enables training object detectors with no human-provided bounding-box annotations.
1. A method for object detection within an image, the method comprising:
obtaining, via a data interface, the image having one or more regions and a caption associated with the image;
encoding, by an image encoder, the image into a visual embedding;
encoding, by a text encoder, at least one word from the caption into a text embedding;
generating, via a cross-attention layer, multimodal features of the image and the word by applying cross-attention between the visual embedding and the text embedding;
computing an activation map based on a gradient of a scalar that represents a similarity between the image and the caption with respect to cross-attention scores of the cross-attention layer;
determining a bounding-box annotation of the word based on the activation map; and
incorporating the bounding-box annotation with the image as a training image sample in a training dataset.
2. The method of claim 1 , wherein computing the activation map comprises averaging values from all attention heads of the cross-attention layer.
3. The method of claim 1 , further comprising:
training an open vocabulary object detector using the training dataset comprising the bounding-box annotation with the image.
4. The method of claim 3 , further comprising:
fine-tuning the open vocabulary object detector using categories trained with human-annotated bounding-boxes.
5. The method of claim 1 , further comprising:
determining the bounding-box annotation based on an overlap between a proposed bounding-box and a relevant region of the activation map.
6. The method of claim 5 , wherein the proposed bounding-box is generated by a pre-trained proposal generator without reference to the caption.
7. The method of claim 1 , further comprising:
training a machine learning model for object detection based on the training image sample having the bounding-box annotation as a ground truth.
8. A system for object detection within an image, the system comprising:
a memory that stores a dialogue structure extraction model;
a communication interface that obtains the image having one or more regions and a caption associated with the image; and
one or more hardware processors that:
encodes, by an image encoder, the image into a visual embedding;
encodes, by a text encoder, at least one word from the caption into a text embedding;
generates, via a cross-attention layer, multimodal features of the image and the word by applying cross-attention between the visual embedding and the text embedding;
computes an activation map based on a gradient of a scalar that represents a similarity between the image and the caption with respect to cross-attention scores of the cross-attention layer;
determines a bounding-box annotation of the word based on the activation map; and
incorporates the bounding-box annotation with the image as a training image sample in a training dataset.
9. The system of claim 8 , wherein the one or more hardware processors computes the activation map by averaging values from all attention heads of the cross-attention layer.
10. The system of claim 8 , wherein the one or more hardware processors further:
trains an open vocabulary object detector using the training dataset comprising the bounding-box annotation with the image.
11. The system of claim 10 , wherein the one or more hardware processors further:
fine-tunes the open vocabulary object detector using categories trained with human-annotated bounding-boxes.
12. The system of claim 8 , wherein the one or more hardware processors further:
determines the bounding-box annotation based on an overlap between a proposed bounding-box and a relevant region of the activation map.
13. The system of claim 12 , wherein the one or more hardware processors generates the proposed bounding-box by a pre-trained proposal generator without reference to the caption.
14. The system of claim 8 , wherein the one or more hardware processors further:
trains a machine learning model for object detection based on the training image sample having the bounding-box annotation as a ground truth.
15. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for object detection within an image, the instructions being executed by a processor to perform operations comprising:
obtaining, via a data interface, the image having one or more regions and a caption associated with the image;
encoding, by an image encoder, the image into a visual embedding;
encoding, by a text encoder, at least one word from the caption into a text embedding;
generating, via a cross-attention layer, multimodal features of the image and the word by applying cross-attention between the visual embedding and the text embedding;
computing an activation map based on a gradient of a scalar that represents a similarity between the image and the caption with respect to cross-attention scores of the cross-attention layer;
determining a bounding-box annotation of the word based on the activation map; and
incorporating the bounding-box annotation with the image as a training image sample in a training dataset.
16. The processor-readable non-transitory storage medium of claim 15 , wherein computing the activation map comprises averaging values from all attention heads of the cross-attention layer.
17. The processor-readable non-transitory storage medium of claim 15 , the instructions being executed by the processor to perform operations further comprising:
training an open vocabulary object detector using the training dataset comprising the bounding-box annotation with the image.