Transformer-based object detection
Object detection using a transformer-based object detection model includes randomly initializing basic queries for the model, modulating the basic queries based on semantics of input images, and training the model basic on features extracted from input images and the modulated queries.
1. A method of detecting objects using a transformer-based object detection model, comprising:
generating modulated queries based on basic queries of a transformer-based object detection model and extracted image features of an input image;
training the transformer-based object detection model using the basic queries and the modulated queries;
replacing the basic queries with the modulated queries as input for a transformer decoder that is to execute the trained transformer-based object detection model; and
inputting the modulated queries and the extracted image features to the transformer decoder.
2. The method of claim 1 , wherein the generating of the modulated queries includes generating convex combinations of the basic queries to produce the modulated queries.
3. The method of claim 2 , wherein the generating of the convex combinations is based on combination coefficients generated by inputting global features of the extracted image features to a multi-layer perceptron (MLP).
4. The method of claim 1 , wherein the extracted image features include image features extracted from input images.
5. A non-volatile computer-readable medium that, when executed, causes at least one processor to perform operations related to object detection comprising:
receiving an input image;
generating dynamic detection queries based on basic queries and semantics of the input image;
training a transformer-based object detection model using the basic queries and the dynamic detection queries; and
performing object detection based on the dynamic detection queries using the trained transformer-based object detection model.
6. The non-volatile computer-readable medium of claim 5 , wherein the generating of the dynamic detection queries comprises:
calculating a global feature as a global average pooling of a feature map of the semantics of the input image;
generating combination coefficients by inputting the global feature to a multi-layer perceptron (MLP); and
performing a convex calculation of the combination coefficients and the basic queries to generate the dynamic detection queries.
7. The non-volatile computer-readable medium of claim 6 , wherein the calculating is based on features extracted from input images.
8. The non-volatile computer-readable medium of claim 5 , wherein the performing of object detection using the trained transformer-based object detection model includes replacing the basic queries with modulated queries.
9. The non-volatile computer-readable medium of claim 5 , wherein the performing of the object detection is by a DETR transformer decoder.
10. The non-volatile computer-readable medium of claim 5 , wherein the at least one processor executes on a server corresponding to a social media platform.
11. The non-volatile computer-readable medium of claim 5 , wherein the at least one processor executes upon a smart device.
12. An apparatus comprising:
at least one processor; and
at least one computer-readable medium that, when executed, causes the at least one processor to perform operations comprising:
extracting features from an input image;
calculating combination coefficients using output from the features;
generating dynamic detection queries as a function of the combination coefficients and basic queries for a transformer-based object detection model; and
performing object detection based on the features and the dynamic detection queries.
13. The apparatus of claim 12 , wherein the apparatus executes a DETR object detection model.
14. The apparatus of claim 12 , wherein calculating the combination coefficients comprise:
receiving the features extracted from the input image;
calculating a global feature of the extracted images; and
inputting the global feature to a multi-layer perceptron (MLP) to generate the combination coefficients.
15. The apparatus of claim 14 , wherein the dynamic detection queries are generated by performing a convex calculation of the combination coefficients and the basic queries.
16. The apparatus of claim 12 , wherein inputs for performing the object detection excludes the basic queries.