Systems and methods for visual question answering
Embodiments described herein provide a zero-shot visual question answering (VQA) framework, which conjoins foundation network models with zero additional training. A first image and a question relating to the first image are received. The first image is divided into a plurality of image patches. A plurality of relevant image patches that are relevant to the question are determined, using a first neural network model, from the plurality of image patches. A plurality of image captions are generated, using a second neural network model, based on the plurality of relevant image patches. An answer to the question is generated based on the plurality of image captions.
1. A method of zero-shot visual question answering, the method comprising:
receiving, via a data interface, a first image and a question relating to the first image;
dividing the first image into a plurality of image patches;
determining, using a first neural network model, relevance of each image patch of the plurality of image patches to the question;
generating, using a second neural network model, a plurality of image captions based on the relevance of each image patch to the question, wherein the generating the plurality of image captions includes:
sampling the plurality of image patches to generate a plurality of sampled image patches based on the relevance of each image patch; and
generating an image caption of the plurality of image captions based on the plurality of sampled image patches; and
generating, using a third neural network model, an answer in response to an input of the question and the plurality of image captions.
2. The method of claim 1 , wherein prior to receiving the first image and the question, each of the first neural network model, the second neural network model, and the third neural network model is pretrained using a separate training dataset.
3. The method of claim 1 , wherein the relevance of each image patch is determined based on a cross-attention score matrix of the plurality of image patches and the question.
4. The method of claim 1 , wherein the second neural network model uses stochastic decoding for generating the plurality of captions.
5. The method of claim 1 , wherein the third neural network model includes a question-answering encoder-decoder model.
6. The method of claim 1 , wherein the generating the answer further comprises:
for each image caption, concatenating the image caption with the question to generate a question and caption combination of a plurality of question and caption combinations;
encoding each of the plurality of question and caption combinations to generate a plurality of corresponding encoded representation;
combining the plurality of encoded representations to provide a concatenated encoded representation; and
generating the answer by decoding the concatenated encoded representation.
7. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising:
receiving, via a data interface, a first image and a question relating to the first image;
dividing the first image into a plurality of image patches;
determining, using a first neural network model, relevance of each image patch of the plurality of image patches to the question;
generating, using a second neural network model, a plurality of image captions based on the relevance of each image patch to the question; and
generating, using a third neural network model, an answer in response to an input of the question and the plurality of image captions, wherein the generating the answer further comprises:
for each image caption, concatenating the image caption with the question to generate a question and caption combination of a plurality of question and caption combinations;
encoding each of the plurality of question and caption combinations to generate a plurality of corresponding encoded representation;
combining the plurality of encoded representations to provide a concatenated encoded representation; and
generating the answer by decoding the concatenated encoded representation.
8. The non-transitory machine-readable medium of claim 7 , wherein prior to receiving the first image and the question, each of the first neural network model, the second neural network model, and the third neural network model is pretrained using a separate training dataset.
9. The non-transitory machine-readable medium of claim 7 , wherein the relevance of each image patch is determined based on a cross-attention score matrix of the plurality of image patches and the question.
10. The non-transitory machine-readable medium of claim 7 , wherein the generating the plurality of image captions includes:
sampling the plurality of image patches to generate a plurality of sampled image patches based on the relevance of each image patch; and
generating an image caption of the plurality of image captions based on the plurality of sampled image patches.
11. The non-transitory machine-readable medium of claim 7 , wherein the second neural network model uses stochastic decoding for generating the plurality of captions.
12. The non-transitory machine-readable medium of claim 7 , wherein the third neural network model includes a question-answering encoder-decoder model.
13. A system, comprising:
a non-transitory memory; and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method comprising:
receiving, via a data interface, a first image and a question relating to the first image;
dividing the first image into a plurality of image patches;
determining, using a first neural network model, relevance of each image patch of the plurality of image patches to the question;
generating, using a second neural network model, a plurality of image captions based on the relevance of each image patch to the question, wherein the generating the plurality of image captions includes:
sampling the plurality of image patches to generate a plurality of sampled image patches based on the relevance of each image patch; and
generating an image caption of the plurality of image captions based on the plurality of sampled image patches; and
generating, using a third neural network model, an answer in response to an input of the question and the plurality of image captions.
14. The system of claim 13 , wherein prior to receiving the first image and the question, each of the first neural network model, the second neural network model, and the third neural network model is pretrained using a separate training dataset.
15. The system of claim 13 , wherein the relevance of each image patch is determined based on a cross-attention score matrix of the plurality of image patches and the question.
16. The system of claim 13 , wherein the second neural network model uses stochastic decoding for generating the plurality of captions.
17. The system of claim 13 , wherein the generating the answer further comprises:
for each image caption, concatenating the image caption with the question to generate a question and caption combination of a plurality of question and caption combinations;
encoding each of the plurality of question and caption combinations to generate a plurality of corresponding encoded representation;
combining the plurality of encoded representations to provide a concatenated encoded representation; and
generating the answer by decoding the concatenated encoded representation.