Method for training visual grounding model by using pseudo label of image and learning device using the same

1 . A method for training a visual grounding model by using a pseudo label of an image, comprising steps of:

(a) in response to acquiring, by a learning device, training data including (1) a training image, (2) one or more GT labels including one or more first object bounding boxes and one or more first object category names, each of which corresponds to one or more first objects located within the training image, and (3) one or more pseudo labels including one or more second object bounding boxes and one or more second object category names, each of which corresponds to one or more second objects located within the training image, which are either same objects as the first objects or different objects, wherein the first object category names are coarse-grained category names corresponding to the first objects and the second object category names are fine-grained category names corresponding to the second objects, performing, by the learning device, sub-processes of (i) inputting the training image and the first object category names to the visual grounding model, to thereby allow the visual grounding model to (i−1) generate one or more first text embedding vectors by embedding the first object category names through a text encoder, (i−2) generate an image embedding vector by embedding the training image through a visual encoder, (i−3) generate one or more proposals, which are regions in the training image where objects are predicted to be located, by using the image embedding vector through a detection head, to thereby align each of proposal embedding vectors of the proposals with each of the corresponding first text embedding vectors, and thus generate one or more object detection results of detecting the first objects corresponding to the first object category names in the training image, and (ii) inputting, to the visual grounding model, one or more cropped images acquired by cropping one or more regions in the training image corresponding to said one or more second object bounding boxes of said one or more pseudo labels, to thereby allow the visual grounding model to generate one or more second visual embedding vectors by embedding the cropped images through the visual encoder, and thus generate one or more image captions corresponding to said one or more cropped images by inputting the second visual embedding vectors to a caption decoder; and

(b) performing, by the learning device, a sub-process of generating one or more visual grounding losses by referring to the GT labels and the object detection results, a sub-process of generating one or more caption losses by referring to the image captions and the pseudo labels, and a sub-process of generating one or more contrastive losses by referring to similarities between each of (1) the first text embedding vectors and one or more second text embedding vectors, each of which is respectively acquired by embedding the first object category names of the GT labels and embedding the second object category names of the pseudo labels through the text encoder, and (2) one or more first visual embedding vectors and the second visual embedding vectors, each of which is respectively acquired by embedding one or more regions corresponding to the first object bounding boxes of the GT labels and embedding one or more regions corresponding to the second object bounding boxes of the pseudo labels through the visual encoder, to thereby train the visual grounding model and the caption decoder by using at least part of the visual grounding losses, the caption losses, and the contrastive losses.

2 . The method of claim 1 , wherein, at the step of (b), while generating the contrastive losses by referring to the similarities between (1) each of the first text embedding vectors and the second text embedding vectors and (2) each of the first visual embedding vectors and the second visual embedding vectors, the learning device generates the contrastive losses by using only a subset of the pseudo labels corresponding to a subset of the second object bounding boxes, wherein IOUs (Intersection Over Union) between the subset of the second object bounding boxes and the first object bounding boxes of the GT labels are less than or equal to a preset threshold IOU.

3 . The method of claim 1 , wherein, at the step of (b), the learning device generates the contrastive losses by using additional one or more negative text embedding vectors acquired by embedding one or more negative object category names through the text encoder, wherein the negative object category names are category names that are different from the first object category names and the second object category names.

4 . The method of claim 1 , wherein, at the step of (a), the GT labels include the first object bounding boxes and the first object category names, wherein (i) (i−1) the first object bounding boxes include a (1_1)-st object bounding box to a (1_n)-th object bounding box corresponding to a (1_1)-st object to a (1_n)-th object included in the first objects and (i−2) the first object category names include a (1_1)-st object category name to a (1_n)-th object category name, and the pseudo labels include the second object bounding boxes and the second object category names, wherein (ii) (ii−1) the second object bounding boxes include a (2_1)-st object bounding box to a (2_m)-th object bounding box corresponding to a (2_1)-st object to a (2_m)-th object included in the second objects and (ii−2) the second object category names include a (2_1)-st object category name to a (2_m)-th object category name, and

wherein, at the step of (b), the learning device generates the contrastive losses by referring to similarities between (1) each of a (1_1)-st text embedding vector to a (1_n)-th text embedding vector and a (2_1)-st text embedding vector to a (2_m)-th text embedding vector, each of which is respectively acquired by embedding each of the (1_1)-st object category name to the (1_n)-th object category and the (2_1)-st object category name to the (2_m)-th object category name through the text encoder, and (2) each of a (1_1)-st visual embedding vector to a (1_n)-th visual embedding vector and a (2_1)-st visual embedding vector to a (2_m)-th visual embedding vector, each of which is respectively acquired by embedding each of regions corresponding to the (1_1)-st object bounding box to the (1_n)-th object bounding box and regions corresponding to the (2_1)-st object bounding box to the (2_m)-th object bounding box through the visual encoder.

5 . The method of claim 1 , wherein, at the step of (b), the learning device generates the caption losses either by referring to similarities between all of the second object category names of the pseudo labels and all of the image captions, or by referring to similarities between part of one or more key words, noun phrases, and verbs extracted from each of the second object category names and part of one or more key words, noun phrases, and verbs extracted from each of the image captions.

6 . The method of claim 1 , wherein, at the step of (b), the learning device generates (i) class losses by referring to predicted class names of the first objects and the first object category names of the GT labels, wherein the predicted class names are acquired by referring to the object detection results, and (ii) the visual grounding losses by referring to the class losses and bounding box losses, wherein the bounding box losses are generated by referring to the proposals and the first object bounding boxes of the GT labels.

7 . The method of claim 6 , wherein the learning device generates the class losses by using contrastive losses between (1) the proposal embedding vectors and (2) the first text embedding vectors corresponding to the first object category names.

8 . The method of claim 6 , wherein, at the step of (b), the learning device generates the bounding box losses by using L1 losses and GIOU (Generalized Intersection Over Union) losses, which are generated by referring to the proposals and the first object bounding boxes.

9 . The method of claim 1 , wherein the pseudo labels including the second object bounding boxes corresponding to the second objects of the training image and the second object category names corresponding to the second objects of the training image are generated through a vision foundation model.

10 . A learning device for training a visual grounding model by using a pseudo label of an image, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to perform processes of: (I) in response to acquiring training data including (1) a training image, (2) one or more GT labels including one or more first object bounding boxes and one or more first object category names, each of which corresponds to one or more first objects located within the training image, and (3) one or more pseudo labels including one or more second object bounding boxes and one or more second object category names, each of which corresponds to one or more second objects located within the training image, which are either same objects as the first objects or different objects, wherein the first object category names are coarse-grained category names corresponding to the first objects and the second object category names are fine-grained category names corresponding to the second objects, (i) inputting the training image and the first object category names to the visual grounding model, to thereby allow the visual grounding model to (i−1) generate one or more first text embedding vectors by embedding the first object category names through a text encoder, (i−2) generate an image embedding vector by embedding the training image through a visual encoder, (i−3) generate one or more proposals, which are regions in the training image where objects are predicted to be located, by using the image embedding vector through a detection head, to thereby align each of proposal embedding vectors of the proposals with each of the corresponding first text embedding vectors, and thus generate one or more object detection results of detecting the first objects corresponding to the first object category names in the training image, and (ii) inputting, to the visual grounding model, one or more cropped images acquired by cropping one or more regions in the training image corresponding to said one or more second object bounding boxes of said one or more pseudo labels, to thereby allow the visual grounding model to generate one or more second visual embedding vectors by embedding the cropped images through the visual encoder, and thus generate one or more image captions corresponding to said one or more cropped images by inputting the second visual embedding vectors to a caption decoder; and (II) generating one or more visual grounding losses by referring to the GT labels and the object detection results, generating one or more caption losses by referring to the image captions and the pseudo labels, and generating one or more contrastive losses by referring to similarities between each of (1) the first text embedding vectors and one or more second text embedding vectors, each of which is respectively acquired by embedding the first object category names of the GT labels and embedding the second object category names of the pseudo labels through the text encoder, and (2) one or more first visual embedding vectors and the second visual embedding vectors, each of which is respectively acquired by embedding one or more regions corresponding to the first object bounding boxes of the GT labels and embedding one or more regions corresponding to the second object bounding boxes of the pseudo labels through the visual encoder, to thereby train the visual grounding model and the caption decoder by using at least part of the visual grounding losses, the caption losses, and the contrastive losses.

11 . The learning device of claim 10 , wherein, at the process of (II), while generating the contrastive losses by referring to the similarities between (1) each of the first text embedding vectors and the second text embedding vectors and (2) each of the first visual embedding vectors and the second visual embedding vectors, the processor generates the contrastive losses by using only a subset of the pseudo labels corresponding to a subset of the second object bounding boxes, wherein IOUs (Intersection Over Union) between the subset of the second object bounding boxes and the first object bounding boxes of the GT labels are less than or equal to a preset threshold IOU.

12 . The learning device of claim 10 , wherein, at the process of (II), the processor generates the contrastive losses by using additional one or more negative text embedding vectors acquired by embedding one or more negative object category names through the text encoder, wherein the negative object category names are category names that are different from the first object category names and the second object category names.

13 . The learning device of claim 10 , wherein, at the process of (I), the GT labels include the first object bounding boxes and the first object category names, wherein (i) (i−1) the first object bounding boxes include a (1_1)-st object bounding box to a (1_n)-th object bounding box corresponding to a (1_1)-st object to a (1_n)-th object included in the first objects and (i−2) the first object category names include a (1_1)-st object category name to a (1_n)-th object category name, and the pseudo labels include the second object bounding boxes and the second object category names, wherein (ii) (ii−1) the second object bounding boxes include a (2_1)-st object bounding box to a (2_m)-th object bounding box corresponding to a (2_1)-st object to a (2_m)-th object included in the second objects and (ii−2) the second object category names include a (2_1)-st object category name to a (2_m)-th object category name, and

wherein, at the process of (II), the processor generates the contrastive losses by referring to similarities between (1) each of a (1_1)-st text embedding vector to a (1_n)-th text embedding vector and a (2_1)-st text embedding vector to a (2_m)-th text embedding vector, each of which is respectively acquired by embedding each of the (1_1)-st object category name to the (1_n)-th object category and the (2_1)-st object category name to the (2_m)-th object category name through the text encoder, and (2) each of a (1_1)-st visual embedding vector to a (1_n)-th visual embedding vector and a (2_1)-st visual embedding vector to a (2_m)-th visual embedding vector, each of which is respectively acquired by embedding each of regions corresponding to the (1_1)-st object bounding box to the (1_n)-th object bounding box and regions corresponding to the (2_1)-st object bounding box to the (2_m)-th object bounding box through the visual encoder.

14 . The learning device of claim 10 , wherein, at the process of (II), the processor generates the caption losses either by referring to similarities between all of the second object category names of the pseudo labels and all of the image captions, or by referring to similarities between part of one or more key words, noun phrases, and verbs extracted from each of the second object category names and part of one or more key words, noun phrases, and verbs extracted from each of the image captions.

15 . The learning device of claim 10 , wherein, at the process of (II), the processor generates (i) class losses by referring to predicted class names of the first objects and the first object category names of the GT labels, wherein the predicted class names are acquired by referring to the object detection results, and (ii) the visual grounding losses by referring to the class losses and bounding box losses, wherein the bounding box losses are generated by referring to the proposals and the first object bounding boxes of the GT labels.

16 . The learning device of claim 15 , wherein the processor generates the class losses by using contrastive losses between (1) the proposal embedding vectors and (2) the first text embedding vectors corresponding to the first object category names.

17 . The learning device of claim 15 , wherein, at the process of (II), the processor generates the bounding box losses by using L1 losses and GIOU (Generalized Intersection Over Union) losses, which are generated by referring to the proposals and the first object bounding boxes.

18 . The learning device of claim 10 , wherein the pseudo labels including the second object bounding boxes corresponding to the second objects of the training image and the second object category names corresponding to the second objects of the training image are generated through a vision foundation model.