Mining unlabeled images with vision and language models for improving object detection
A method for object detection obtains, from a set of RGB images lacking annotations, a set of regions that include potential objects, a bounding box, and an objectness score indicating a region prediction confidence. The method obtains, by a region scorer for each region in the set, a category from a fixed set of categories and a confidence for the category responsive to the objectness score. The method duplicates each region in the set to obtain a first and a second patch. The method encodes the patches to obtain an image vector. The method encodes a template sentence using the category to obtain a text vector for each category. The method compares the image vector to the text vector via a similarity function to obtain a similarity probability based on the confidence. The method defines a final set of pseudo labels based on the similarity probability being above a threshold.
1 . A computer-implemented method for object detection, comprising:
obtaining, from a set of RGB images lacking annotations, a set of regions in an RGB image that include one or more potential objects, a bounding box for the one or more potential objects, and an objectness score indicating a region prediction confidence;
obtaining, by a region scorer for each region in the set, a category from a fixed set of categories and a confidence for the category name calculated as:
S
i
u
=
S
RPN
(
R
i
)
+
max
(
p
i
u
)
2
where S RPN (R i ) is the objectness score for a region R i and p i u is a probability distribution for the category name;
duplicating each region in the set to obtain a first and a second image patch;
encoding, by a visional and language (V & L) image encoder, the first and the second image patches to obtain an image vector;
encoding, by a V & L text encoder, a template sentence using the category to obtain a text vector for each category having a same dimensionality as the image vector;
comparing the image vector to the text vector via a similarity function to obtain a similarity probability based on the confidence; and
defining a final set of pseudo labels based on the similarity probability being above a user-defined similarity probability threshold.
2 . The computer-implemented method of claim 1 , further comprising discarding any regions equal to and below the user-defined threshold probability.
3 . The computer-implemented method of claim 1 , further comprising duplicating each region using into an original scale region and a larger scale region having same bounding box centers to obtain the first and the second image patch.
4 . The computer-implemented method of claim 3 , wherein the larger scale region is 1.5× larger than the original scale region.
5 . The computer-implemented method of claim 1 , further comprising rescaling the first and the second image patch to obtain a rescaled first and second image patches fitting an input of the vision and language encoder.
6 . The computer-implemented method of claim 1 , further comprising applying a softmax function over similarities between the image vector and the text vector to obtain a probability between 0 and 1 for a region to belong to a particular category c.
7 . The computer-implemented method of claim 1 , wherein a dimensionality of the image vector is defined by a V & L model comprising the V & L image and text encoders, the V & L model trained on pairs of images and corresponding captions.
8 . The computer-implemented method of claim 1 , wherein the image vector and the text vector have a same dimensionality.
9 . The computer-implemented method of claim 1 , wherein a maximum similarity probability is averaged with the confidence score, the confidence score having a value between 0 and 1.
10 . The computer-implemented method of claim 1 , further comprising training an object detection system with a combination of labeled data and the pseudo labels for open-vocabulary detection.
11 . The computer-implemented method of claim 1 , further comprising cropping the first and the second image patch from the RGB image prior to an encoding.
12 . The computer-implementing method of claim 1 , further comprising automatically controlling a vehicle system for collision avoidance responsive to a pseudo label predicting an impending collision.
13 . A computer program product for object detection, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
obtaining, by a hardware processor of the computer from a set of ROB images lacking annotations, a set of regions in an RGB image that include one or more potential objects, a bounding box for the one or more potential objects, and an objectness score indicating a region prediction confidence;
obtaining, by a region scorer implemented by the hardware processor for each region in the set, a category from a fixed set of categories and a confidence for the category name calculated as:
S
i
u
=
S
RPN
(
R
i
)
+
max
(
p
i
u
)
2
where S RPN (R i ) is the objectness score for a region R i and p i u is a probability distribution for the category name;
duplicating, by the hardware processor, each region in the set to obtain a first and a second image patch;
encoding, by a visional and language (V& L) image encoder implemented by the hardware processor, the first and the second image patches to obtain an image vector;
encoding, by a V&L text encoder implemented by the hardware processor, a template sentence using the category to obtain a text vector for each category having a same dimensionality as the image vector;
comparing, by the hardware processor, the image vector to the text vector via a similarity function to obtain a similarity probability based on the confidence; and
defining, by the hardware processor, a final set of pseudo labels based on the similarity probability being above a user-defined similarity probability threshold.
14 . The computer program product of claim 13 , wherein the method further comprises discarding any regions equal to and below the user-defined threshold probability.
15 . The computer program product of claim 13 , wherein the method further comprises duplicating each region using into an original scale region and a larger scale region having same bounding box centers to obtain the first and the second image patch.
16 . The computer program product of claim 15 , wherein the larger scale region is 1.5× larger than the original scale region.
17 . The computer program product of claim 13 , wherein the method further comprises rescaling the first and the second image patch to obtain a rescaled first and second image patches fitting an input of the vision and language encoder.
18 . The computer program product of claim 13 , wherein the method further comprises applying a softmax function over similarities between the image vector and the text vector to obtain a probability between 0 and 1 for a region to belong to a particular category c.
19 . The computer program product of claim 13 , wherein a dimensionality of the image vector is defined by a V & L model comprising the V & L image and text encoders, the V & L model trained on pairs of images and corresponding captions.
20 . A computer processing system fir object detection, comprising:
a memory device for storing program code; and
a hardware processor operatively coupled to the memory device for storing program code to:
obtain, from a set of RGB images lacking annotations, a set of regions in an RGB image that include one or more potential objects, a bounding box for the one or more potential objects, and an objectness score indicating a region prediction confidence
obtain, for each region in the set, a category from a fixed set of categories and a confidence for the category name calculated as:
S
i
u
=
S
RPN
(
R
i
)
+
max
(
p
i
u
)
2
where S RPN (R i ) is the objectness score for a region R i and p i u is a probability distribution for the category name;
duplicate each region in the set to obtain a first and a second image patch;
encode, by a visional and language (V& L) image encoder implemented by the hardware processor, the first and the second image patches to obtain an image vector;
encode, by a V&L text encoder implemented by the hardware processor, a template sentence using the category to obtain a text vector for each category having a same dimensionality as the image vector;
compare the image vector to the text vector via a similarity function to obtain a similarity probability based on the confidence; and
define a final set of pseudo labels based on the similarity probability being above a user-defined similarity probability threshold.