Multi-granularity alignment for visual question answering
In one embodiment, a method includes accessing an image and a natural-language question regarding the image and extracting, from the image, a first set of image features at a first level of granularity and a second set of image features at a second level of granularity. The method further includes extracting, from the question, a first set of text features at the first level of granularity and a second set of text features at the second level of granularity; generating a first output representing an alignment between the first set of image features and the first set of text features; generating a second output representing an alignment between the second set of image features and the second set of text features; and determining an answer to the question based on the first output and the second output.
1. A method comprising:
accessing an image and a natural-language question regarding the image;
extracting, from the image:
a first set of image features at a first, spatial level of granularity comprising a set of first image portions, each first image portion having a first number of pixels; and
a second set of image features at a second, regional level of granularity comprising a set of one or more second image portions, each second image portion containing one or more objects in the image and having a second number of pixels that is greater than the first number of pixels;
extracting, from the question:
a first set of text features at a first level of granularity that separately identifies each word in the natural-language question;
and a second set of text features at a second level of granularity that separately identifies one or more phrases in the natural-language question;
generating a first output representing an alignment between the first set of image features at the first, spatial level of granularity and the first set of text features at the first level of granularity;
generating a second output representing an alignment between the second set of image features at the second, regional level of granularity and the second set of text features at the second level of granularity; and
determining an answer to the question based on the first output and the second output.
2. The method of claim 1 , wherein each image feature in the second set of image features corresponds to one or more object regions in the image.
3. The method of claim 2 , wherein each text feature in the second set of text features corresponds to a noun phrase in the natural-language question.
4. The method of claim 3 , further comprising:
extracting, from the image, a third set of image features at a third level of granularity;
extracting, from the question, a third set of text features at the third level of granularity;
generating a third output representing an alignment between the third set of image features at the third level of granularity and the third set of text features at the third level of granularity; and
determining an answer to the question based on the first output, the second output, and the third output.
5. The method of claim 4 , wherein each image feature in the third set of image features corresponds to a concept from a set of concepts, each concept providing a semantic representation of at least a portion of the image.
6. The method of claim 5 , wherein each text feature in the third set of text features corresponds to an entity from a set of entities, each entity corresponding to an object derived from a noun phrase in the natural-language question.
7. The method of claim 1 , further comprising tokenizing the image features into image token feature vectors and tokenizing the text features into text token feature vectors.
8. The method of claim 7 , wherein:
each image token feature vector corresponds to a node in a graph representing the image at the corresponding level of granularity; and
each text token feature vector corresponds to a node in a graph representing the natural-language question at the corresponding level of granularity.
9. The method of claim 8 , further comprising:
generating, from the image and at each level of granularity, an image lead graph; and
generating, from the natural-language questions and at each level of granularity, a text lead graph.
10. The method of claim 9 , wherein:
generating the first output comprises determining, by a transformer model, the first output based on: the image token feature vector at the first level of granularity, the text token feature vector at the first level of granularity, the image lead graph at the first level of granularity, and the text lead graph at the first level of granularity; and
generating the second output comprises determining, by the transformer model, the second output based on: the image token feature vector at the second level of granularity, the text token feature vector at the second level of granularity, the image lead graph at the second level of granularity, and the text lead graph at the second level of granularity.
11. The method of claim 10 , further comprising concatenating, by the transformer model, the image token feature vector and the text token feature vector at each level of granularity.
12. The method of claim 11 , further comprising:
merging, by the transformer model, the image lead graph and the text lead graph at each level of granularity; and
modifying, using the merged lead graphs at each level, the concatenated image token feature vector and text token feature vector at that level.
13. The method of claim 12 , wherein the transformer model comprises a GA-TRM transformer.
14. The method of claim 1 , wherein determining the answer to the question comprises selecting an answer from a set of candidate answers based on a probability associated with each candidate answer.
15. One or more non-transitory computer readable storage media embodying instructions and coupled to one or more processors that are operable to execute the instructions to:
access an image and a natural-language question regarding the image;
extract, from the image:
a first set of image features at a first, spatial level of granularity comprising a set of first image portions, each first image portion having a first number of pixels; and
a second set of image features at a second, regional level of granularity comprising a set of one or more second image portions, each second image portion containing one or more objects in the image and having a second number of pixels that is greater than the first number of pixels;
extract, from the question:
a first set of text features at a first level of granularity that separately identifies each word in the natural-language question;
and a second set of text features at a second level of granularity that separately identifies one or more phrases in the natural-language question;
generate a first output representing an alignment between the first set of image features at the first, spatial level of granularity and the first set of text features at the first level of granularity;
generate a second output representing an alignment between the second set of image features at the second, regional level of granularity and the second set of text features at the second level of granularity; and
determine an answer to the question based on the first output and the second output.
16. The media of claim 15 , wherein the one or more processors are further operable to execute the instructions to:
extract, from the image, a third set of image features at a third level of granularity;
extract, from the question, a third set of text features at the third level of granularity;
generate a third output representing an alignment between the third set of image features at the third level of granularity and the third set of text features at the third level of granularity; and
determine an answer to the question based on the first output, the second output, and the third output.
17. A system comprising one or more processors and a non-transitory computer readable storage media embodying instructions coupled to the one or more processors, the one or more processors operable to execute the instructions to:
access an image and a natural-language question regarding the image;
extract, from the image:
a first set of image features at a first, spatial level of granularity comprising a set of first image portions, each first image portion having a first number of pixels; and
a second set of image features at a second, regional level of granularity comprising a set of one or more second image portions, each second image portion containing one or more objects in the image and having a second number of pixels that is greater than the first number of pixels;
extract, from the question;
a first set of text features at a first level of granularity that separately identifies each word in the natural-language question;
and a second set of text features at a second level of granularity that separately identifies one or more phrases in the natural-language question;
generate a first output representing an alignment between the first set of image features at the first, spatial level of granularity and the first set of text features at the first level of granularity;
generate a second output representing an alignment between the second set of image features at the second, regional level of granularity and the second set of text features at the second level of granularity; and
determine an answer to the question based on the first output and the second output.
18. The system of claim 17 , wherein the one or more processors are further operable to execute the instructions to:
extract, from the image, a third set of image features at a third level of granularity;
extract, from the question, a third set of text features at the third level of granularity;
generate a third output representing an alignment between the third set of image features at the third level of granularity and the third set of text features at the third level of granularity; and
determine an answer to the question based on the first output, the second output, and the third output.
19. The system of claim 18 , wherein each image feature in the third set of image features corresponds to a concept from a set of concepts, each concept providing a semantic representation of at least a portion of the image.
20. The system of claim 17 , further comprising one or more processors that are operable to execute the instructions to tokenize the image features into image token feature vectors and tokenizing the text features into text token feature vectors.