OPEN-VOCABULARY OBJECT DETECTION WITH VISION AND LANGUAGE SUPERVISION
A computer-implemented method for training a neural network to predict object categories without manual annotation is provided. The method includes feeding training datasets including at least images and data annotations to an object detection neural network, converting, by a text prompter, the data annotations into natural text inputs, converting, by a text embedder, the natural text inputs into embeddings, minimizing objective functions during training to adjust parameters of the object detection neural network, and predicting, by the object detection neural network, objects within images and videos.
1 . A computer-implemented method for training a neural network to predict object categories without manual annotation, the method comprising:
feeding training datasets including at least images and data annotations to an object detection neural network;
converting, by a text prompter, the data annotations into natural text inputs;
converting, by a text embedder, the natural text inputs into embeddings;
minimizing objective functions during training to adjust parameters of the object detection neural network; and
predicting, by the object detection neural network, objects within images and videos.
2 . The computer-implemented method of claim 1 , wherein the training datasets include at least detection data, image caption data, attribute data, and action data.
3 . The computer-implemented method of claim 2 , wherein the detection data is converted into a simple sentence concatenating all object categories.
4 . The computer-implemented method of claim 1 , wherein each object is described with a bounding box and some semantic description.
5 . The computer-implemented method of claim 1 , wherein localization is applied to minimize localization ability of the object detection neural network.
6 . The computer-implemented method of claim 1 , wherein a classification loss is applied when category annotations are available for each object.
7 . The computer-implemented method of claim 1 , wherein a weak alignment objective function and a global alignment objective function are applied, the weak alignment objective function trained from image-caption pairs without localization annotation.
8 . A computer program product for training a neural network to predict object categories without manual annotation, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
feeding training datasets including at least images and data annotations to an object detection neural network;
converting, by a text prompter, the data annotations into natural text inputs;
converting, by a text embedder, the natural text inputs into embeddings;
minimizing objective functions during training to adjust parameters of the object detection neural network; and
predicting, by the object detection neural network, objects within images and videos.
9 . The computer program product of claim 8 , wherein the training datasets include at least detection data, image caption data, attribute data, and action data.
10 . The computer program product of claim 9 , wherein the detection data is converted into a simple sentence concatenating all object categories.
11 . The computer program product of claim 8 , wherein each object is described with a bounding box and some semantic description.
12 . The computer program product of claim 8 , wherein localization is applied to minimize localization ability of the object detection neural network.
13 . The computer program product of claim 8 , wherein a classification loss is applied when category annotations are available for each object.
14 . The computer program product of claim 8 , wherein a weak alignment objective function and a global alignment objective function are applied, the weak alignment objective function trained from image-caption pairs without localization annotation.
15 . A computer processing system for training a neural network to predict object categories without manual annotation, comprising:
a memory device for storing program code; and
a processor device, operatively coupled to the memory device, for running the program code to:
feed training datasets including at least images and data annotations to an object detection neural network;
convert, by a text prompter, the data annotations into natural text inputs;
convert, by a text embedder, the natural text inputs into embeddings;
minimize objective functions during training to adjust parameters of the object detection neural network; and
predict, by the object detection neural network, objects within images and videos.
16 . The computer processing system of claim 15 , wherein the training datasets include at least detection data, image caption data, attribute data, and action data.
17 . The computer processing system of claim 15 , wherein each object is described with a bounding box and some semantic description.
18 . The computer processing system of claim 15 , wherein localization is applied to minimize localization ability of the object detection neural network.
19 . The computer processing system of claim 15 , wherein a classification loss is applied when category annotations are available for each object.
20 . The computer processing system of claim 15 , wherein a weak alignment objective function and a global alignment objective function are applied, the weak alignment objective function trained from image-caption pairs without localization annotation.