Training visual language grounding models using separation loss

1. A system, comprising a processor to:

receive a randomly generated alpha-map, a pair of training images, and a pair of training texts associated with the pair of training images;

generate a blended image based on the randomly generated alpha-map and the pair of training images; and

train a visual language grounding model to separate the blended image into a pair of heatmaps identifying portions of the blended image corresponding to each of the training images using a separation loss.

2. The system of claim 1 , wherein the training texts comprise natural free-form texts.

3. The system of claim 1 , wherein the visual language grounding model comprises an encoder to generate image encodings based on the alpha-map and the pair of training images, a text conditioner to generate a plurality of text attenuated image encodings based on the image encodings and the pair of training texts, and a decoder to convert the text attenuated image encodings into heatmaps.

4. The system of claim 3 , wherein the text conditioner comprises a Bidirectional Encoder Representations from Transformers (BERT) model.

5. The system of claim 4 , wherein the text conditioner comprises a plurality of projection modules coupled to the BERT model.

6. The system of claim 1 , wherein the visual language grounding model is trained using an unconditioned adversary loss.

7. The system of claim 1 , comprising a separately trained detector-based weak supervised grounding network, wherein the separately trained detector-based WSG network is to generate bounding boxes scores based on a received image and the trained visual language grounding model is to generate a first heatmap based on the received image, wherein the bounding box scores are converted to a second heatmap using assignment of the bounding box scores to pixels of the bounding box, and wherein the first heatmap and the second heatmap are averaged to generate a combined heatmap.

8. A computer-implemented method, comprising:

receiving, via a processor, training images, corresponding training texts, and a randomly generated alpha-map;

combining, via the processor, a pair of the training images using the randomly generated alpha-map to generate a blended image; and

training, via the processor, a visual language grounding model to separate the training images in the blended image as conditioned on the corresponding training texts.

9. The computer-implemented method of claim 8 , comprising:

receiving, via the processor, an image and a query phrase;

inputting, via the processor, the image and the query phrase into the trained visual language grounding model; and

receiving, via the processor, an output heatmap for the query phrase from the trained visual language grounding model.

10. The computer-implemented method of claim 8 , wherein training the visual language grounding model comprises calculating a separation loss for each of the pair of training images as a main training objective.

11. The computer-implemented method of claim 10 , wherein training the visual language grounding model comprises calculating an image-to-text loss for text and image feature distribution alignment.

12. The computer-implemented method of claim 8 , wherein training the visual language grounding model comprises calculating a negative texts loss based on a third received training text that is unrelated to the pair of training images.

13. The computer-implemented method of claim 8 , wherein training the visual language grounding model comprises calculating an unconditioned adversary loss to decrease overfitting on artifacts.

14. The computer-implemented method of claim 8 , further comprising:

receiving an image and a query phrase;

inputting the image and the query phrase into the trained visual language grounding model and a separately trained detector-based model;

receiving a first heatmap associated with the image for the query phrase from the trained visual language grounding model;

receiving a bounding box with a bounding box score for the query from the separately trained detector-based model;

converting the bounding box score into assigned values for pixels in the bounding box to generate a second heatmap for the image; and

combining the first heatmap and the second heatmap to generate a combined heatmap for the query and the image.

15. A computer program product for training visual language grounding models, the computer program product comprising a computer-readable storage medium having program code embodied therewith, wherein the computer-readable storage medium is not a signal per se, the program code executable by a processor to cause the processor to:

receive training images, corresponding training texts, and a randomly generated alpha-map;

combine a pair of the training images using the randomly generated alpha-map to generate a blended image; and

train a visual language grounding model to separate the training images in the blended image as conditioned on the corresponding training texts.

16. The computer program product of claim 15 , further comprising program code executable by the processor to;

receive an image and a query phrase;

input the image and the query phrase into the trained visual language grounding model; and

receive an output heatmap for the query phrase from the trained visual language grounding model.

17. The computer program product of claim 15 , further comprising program code executable by the processor to calculate a separation loss for each of the pair of training images as a main training objective.

18. The computer program product of claim 15 , further comprising program code executable by the processor to calculate an image-to-text loss for text and image feature distribution alignment.

19. The computer program product of claim 15 , further comprising program code executable by the processor to calculate a negative texts loss based on a third received training text that is unrelated to the pair of training images.

20. The computer program product of claim 15 , further comprising program code executable by the processor to calculate an unconditioned adversary loss to decrease overfitting on artifacts.