Contextual grounding of natural language phrases in images

1. A method for text-image retrieval including text and image branches, said method comprising:

receiving as input a text query and an image;

parsing the input text query into tokens and converting them to entity embedding vectors;

locating visual object candidates in the input image;

scoring correspondences between the entity embeddings and visual object candidates;

providing, visualized in a bounding box, the object corresponding to the query text entity with the highest probability score, to a user of the system;

pre-training the text branch utilizing a BERT, Bidirectional Encoder Representations from Transformers, base model;

receiving, by the image branch, region of interest (RoI) features as input objects from an object detector;

training, a two-layer multi-layer perceptron (MLP) to generate spatial embedding given absolute spatial information of the RoI location and size normalized to the entire image;

adding, by both branches, positional and spatial embedding to tokens and RoIs respectively as input to a first interaction layer of the MLP; and

performing, at each layer of the MLP, self-attenuation by each hidden representation to each other to generate a new hidden representation as layer output;

wherein no specific embedding or object feature extraction is used in the method.

2. The method of claim 1 further comprising:

providing, at the end of each branch, a final hidden state to a ground head to provide cross-modal attention responses with text entity hidden states as queries and image object hidden representations as keys.

3. The method of claim 2 wherein matching correspondences are determined from the attention responses.

4. The method of claim 3 further comprising

back propagating a mean binary cross entropy loss per entity if the correspondence(s0 does not match a ground truth.