Systems and methods for visual question answering using image relevant textual prompts

1 . A method of zero-shot visual question answering, the method comprising:

receiving, via a data interface, a first image and a first question relating to a visual content of the first image;

determining, using an image-question matching model, one or more question-relevant image regions of the first image;

determining, using a visual-language neural network model, one or more image patches of the first image relevant to the first question wherein the image patches are selected based at least in part on a patch-level relevance score derived from the one or more question-relevant image regions;

generating, using a caption model, one or more image captions based on the one or more image patches of the first image;

extracting, using an answer extract model, answer candidates from the one or more image captions;

generating synthetic questions using the answer candidates;

generating synthetic question-answer pairs using the synthetic questions and the answer candidates;

generating a question-answer prompt by concatenating the synthetic question-answer pairs; and

generating, using a first language neural network model, a first answer to the first question using an input of the first question prepended with the question-answer prompt.

2 . The method of claim 1 , further comprising:

generating a caption prompt using the one or more image captions relevant to the first question; and

generating the input to the first language neural network model by concatenating the caption prompt, the question-answer prompt, and the first question.

3 . The method of claim 1 , wherein the generating the synthetic questions using the answer candidates includes:

generating, using question templates, the synthetic questions,

wherein the question templates are determined based on types of the answer candidates.

4 . The method of claim 3 , wherein the types of the answer candidates include at least one or more of nouns, verbs, adjectives, and numbers.

5 . The method of claim 1 , wherein the generating the synthetic questions using the answer candidates includes:

generating, using a second language neural network model, the synthetic questions using the answer candidates.

6 . The method of claim 5 , wherein the second language neural network model is trained on one or more textual question-answer datasets for generating questions from answers.

7 . The method of claim 6 , wherein during the training the second language neural network model, an input to the second language neural network model is generated by concatenating an answer text and a corresponding context text from the question-answer datasets.

8 . A system for providing zero-shot visual question answering, the system comprising:

a memory that stores a zero-shot visual question answering model and a plurality of processor-executable instructions;

a communication interface that receives a first image and a first question relating to a visual content of the first image; and

one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:

determining, using an image-question matching model, one or more question-relevant image regions of the first image;

determining, using a visual-language neural network model, one or more image patches of the first image relevant to the first question wherein the image patches are selected based at least in part on a patch-level relevance score derived from the one or more question-relevant image regions;

generating, using a caption model, one or more image captions based on the one or more image patches of the first image;

generating, using the one or more image captions, answer candidates;

generating synthetic questions using the answer candidates;

generating synthetic question-answer pairs using the synthetic questions and the answer candidates;

generating a question-answer prompt by concatenating the synthetic question-answer pairs; and

generating, using a first language neural network model, a first answer to the first question using an input of the first question prepended with the question-answer prompt.

9 . The system of claim 8 , wherein the operations further comprise:

generating a caption prompt using the one or more image captions relevant to the first question; and

generating the input to the first language neural network model by concatenating the caption prompt, the question-answer prompt, and the first question.

10 . The system of claim 8 , wherein the generating the synthetic questions using the answer candidates includes:

generating, using question templates, the synthetic questions,

wherein the question templates are determined based on types of the answer candidates.

11 . The system of claim 10 , wherein the types of the answer candidates include at least one or more of nouns, verbs, adjectives, and numbers.

12 . The system of claim 8 , wherein the generating the synthetic questions using the answer candidates includes:

generating, using a second language neural network model, the synthetic questions using the answer candidates.

13 . The system of claim 12 , wherein the second language neural network model is trained on one or more textual question-answer datasets for generating questions from answers.

14 . The system of claim 13 , wherein during the training the second language neural network model, an input to the second language neural network model is generated by concatenating an answer text and a corresponding context text from the question-answer datasets.

15 . A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:

receiving, via a data interface, a first image and a first question relating to a visual content of the first image;

determining, using an image-question matching model, one or more question-relevant image regions of the first image;

determining, using a visual-language neural network model, one or more image patches of the first image relevant to the first question wherein the image patches are selected based at least in part on a patch-level relevance score derived from the one or more question-relevant image regions;

generating, using a caption model, one or more image captions based on the one or more image patches of the first image;

generating, using the one or more image captions, answer candidates;

generating synthetic questions using the answer candidates;

generating synthetic question-answer pairs using the synthetic questions and the answer candidates;

generating a question-answer prompt by concatenating the synthetic question-answer pairs; and

generating, using a first language neural network model, a first answer to the first question using an input of the first question prepended with the question-answer prompt.

16 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:

generating a caption prompt using the one or more image captions relevant to the first question; and

generating the input to the first language neural network model by concatenating the caption prompt, the question-answer prompt, and the first question.

17 . The non-transitory machine-readable medium of claim 15 , wherein the generating the synthetic questions using the answer candidates includes:

generating, using question templates, the synthetic questions,

wherein the question templates are determined based on types of the answer candidates.

18 . The non-transitory machine-readable medium of claim 17 , wherein the types of the answer candidates include at least one or more of nouns, verbs, adjectives, and numbers.

19 . The non-transitory machine-readable medium of claim 15 , wherein the generating the synthetic questions using the answer candidates includes:

generating, using a second language neural network model, the synthetic questions using the answer candidates.

20 . The non-transitory machine-readable medium of claim 19 , wherein the second language neural network model is trained on one or more textual question-answer datasets for generating questions from answers.