Align-to-ground, weakly supervised phrase grounding guided by image-caption alignment

1. A method for visual grounding of a caption in an image, the method comprising:

projecting at least two parsed phrases of the caption into a trained semantic embedding space;

projecting extracted region proposals of the image into the trained semantic embedding space;

aligning the extracted region proposals and the at least two parsed phrases;

aggregating the aligned region proposals and the at least two parsed phrases to determine a caption-conditioned image representation; and

projecting the caption-conditioned image representation and the caption into a semantic embedding space to align the caption-conditioned image representation and the caption.

2. The method of claim 1 , further comprising parsing the caption into the at least two parsed phrases.

3. The method of claim 1 , further comprising extracting the region proposals from the image.

4. The method of claim 1 , wherein the extracted region proposals and the at least two parsed phrases are aligned by determining a semantic similarity between the projected extracted region proposals and the projected at least two parsed phrases in the trained semantic embedding space.

5. The method of claim 4 , wherein the semantic similarity is determined using a distance function.

6. The method of 5 , wherein the distance function comprises at least one of a cosine function, a Euclidean function and an L 1 function.

7. The method of claim 1 , wherein the aligned region proposals and the at least two parsed phrases are aggregated using at least one of a set-encoder, a recurrent neural network (RNN), a convolutional neural network (CNN) and graphs.

8. The method of claim 1 , wherein the embedded caption-conditioned image representation and the embedded caption are aligned in the semantic embedding space by computing a distance function between the projected caption-conditioned image representation and the projected caption.

9. The method of claim 1 , wherein projecting at least two parsed phrases of the caption into a trained semantic embedding space comprises:

determining a word vector representation of each of the at least two parsed phrases; and

projecting the word vector representation of each of the at least two parsed phrases into the trained semantic embedding space such that vector representations that are related are closer together in the trained semantic embedding space than unrelated vectors.

10. The method of claim 1 , wherein projecting extracted region proposals of the image into the trained semantic embedding space comprises:

extracting appearance features of each of each of the extracted region proposals;

determining a feature vector representation of each of the extracted region proposals; and

projecting the feature vector representation of each of the extracted region proposals into the trained semantic embedding space such that vector representations that are related are closer together in the trained semantic embedding space than unrelated vectors.

11. The method of claim 10 , wherein the semantic embedding space is created using a loss function.

12. An apparatus for visual grounding of a caption in an image, comprising:

a local matching module to:

project at least two parsed phrases of the caption and extracted region proposals of the image into a trained semantic embedding space; and

aligning the extracted region proposals and the at least two parsed phrases;

a local aggregator module to aggregate the aligned region proposals and the at least two parsed phrases to determine a caption-conditioned image representation; and

a global matching module to project the caption-conditioned image representation and the caption into a semantic embedding space to align the caption-conditioned image representation and the caption.

13. The apparatus of claim 12 , further comprising a parsing module to parse the caption into the at least two parsed phrases.

14. The apparatus of claim 12 , further comprising a region proposal module to extract the region proposals from the image.

15. The apparatus of claim 12 , wherein the local aggregator module aggregates the aligned region proposals and the at least two parsed phrases using a permutation-invariant set-encoder using a two-layer multilayer perceptron (MLP).

16. The apparatus of claim 12 , wherein the global matching module aligns the embedded caption-conditioned image representation and the embedded caption by computing a cosine similarity between the projected caption-conditioned image representation and the projected caption.

17. The apparatus of claim 12 , wherein the local matching module projects the at least two parsed phrases of the caption into a trained semantic embedding space by:

determining a word vector representation of each of the at least two parsed phrases; and

projecting the word vector representation of each of the at least two parsed phrases into the trained semantic embedding space such that vector representations that are related are closer together in the trained semantic embedding space than unrelated vectors.

18. The apparatus of claim 12 , wherein the local matching module projects the extracted region proposals of the image into the trained semantic embedding space by:

extracting appearance features of each of each of the extracted region proposals;

determining a feature vector representation of each of the extracted region proposals; and

projecting the feature vector representation of each of the extracted region proposals into the trained semantic embedding space such that vector representations that are related are closer together in the trained semantic embedding space than unrelated vectors.

19. The apparatus of claim 12 , wherein the apparatus is further configured to create a trained semantic embedding space for use for visual grounding of a caption in an image, the creating a trained semantic embedding space comprising:

for each image region of a plurality of images having known respective phrases, extracting semantic appearance features;

determining a feature vector representation for the extracted features of each of the image regions;

determining a word vector representation for the respective phrases of each of the image regions; and

for each of the image regions, semantically embedding the feature vector representation and the word vector representation for the respective phrases of each of the image regions in a semantic embedding space such that embedded feature vector representations and word vector representations that are related are closer together in the semantic embedding space than unrelated vectors.