Systems and methods for a vision-language pretraining framework
Embodiments described herein provide a multimodal vision-language model. The multimodal vision-language model contains a Generalist Multimodal Transformer capable of complete multiple tasks using the same set of parameters learning from pre-training. The Generalist Multimodal Transformer allows alignment between frozen, unimodal encoders, such as image encoders and large language models. The Generalist Multimodal Transformer eliminates the need for fine-tuning the image encoders and large language models.
1 . A method of a multimodal framework for vision-language tasks, the method comprising:
receiving, at a user interface and at the multimodal framework comprising an image encoder, a query transformer, and a pretrained language model, an input image and an input utterance instructing a task relating to the input image;
encoding, by the image encoder, the input image into an image representation;
transforming, by the query transformer connected to an output of the image encoder, the image representation and a set of query vectors to extract visual features from the image representation into a single transformed representation, wherein the query transformer comprises a self-attention layer to generate a self-attention output from the set of query vectors and a cross-attention layer connected to an output of the self-attention layer and the output of the image encoder to generate a cross-attention output from the image representation and the self-attention output;
generating, by the pretrained language model, a single combined representation from a first input of the single transformed representation from the query transformer and a second input of the input utterance;
generating, by the pretrained language model, a decoded output text based on the single combined representation; and
presenting, via the user interface, a response based on the decoded output text in response to the input utterance.
2 . The method of claim 1 , wherein the input utterance indicates an expected output text to generate based on visual content of the input image.
3 . The method of claim 1 , wherein the pretrained language model includes a text decoder, and wherein the text decoder receives an input which combines the single transformed representation and tokens of the input utterance.
4 . The method of claim 1 , wherein the pretrained language model includes a text encoder and a text decoder, and wherein the single combined representation is generated by:
encoding, via the text encoder, a combination of the single transformed representation, the input text, and a prefix text into the single combined representation;
decoding, via the text decoder, a suffix text from the single combined representation; and
concatenating the prefix text and the suffix text into the decoded output text.
5 . The method of claim 1 , further comprising:
projecting, via a fully connected layer, the single transformed representation to a same dimension with the pretrained language model before feeding the single transformed representation to the pretrained language model.
6 . The method of claim 1 , wherein the decoded output text is generated token by token by the pretrained language model conditioned on previously generated tokens.
7 . The method of claim 1 , wherein the query transformer includes a fewer number of parameters compared to the image encoder or the pretrained language model.
8 . The method of claim 1 , wherein the query transformer is trained with a training dataset of images and accompanying texts while the image encoder and the pretrained language model are kept frozen during training, and
wherein the training dataset of images and accompanying texts does not correspond to a particular vision-language task.
9 . A system of a multimodal framework for vision-language tasks, the system comprising:
a user interface receiving an input image and an input utterance instructing a task relating to the input image;
a memory storing a multimodal framework comprising an image encoder, a query transformer, a pretrained language model and a plurality of processor-executable instructions; and
one or more processors executing the instructions to perform operations including:
encoding, by the image encoder, the input image into an image representation;
transforming, by the query transformer connected to an output of the image encoder, the image representation and a set of query vectors to extract visual features from the image representation into a single transformed representation, wherein the query transformer comprises a self-attention layer to generate a self-attention output from the set of query vectors and a cross-attention layer connected to an output of the self-attention layer and the output of the image encoder to generate a cross-attention output from the image representation and the self-attention output;
generating, by the pretrained language model, a single combined representation from a first input of the single transformed representation from the query transformer and a second input of the input utterance;
generating, by the pretrained language model, a decoded output text based on the single combined representation; and
presenting, via the user interface, a response based on the decoded output text in response to the input utterance.
10 . The system of claim 9 , wherein the input utterance indicates an expected output text to generate based on visual content of the input image.
11 . The system of claim 9 , wherein the pretrained language model includes a text decoder, and wherein the text decoder receives an input which combines the single transformed representation and tokens of the input utterance.
12 . The system of claim 9 , wherein the pretrained language model includes a text encoder and a text decoder, and wherein the single combined representation is generated by:
encoding, via the text encoder, a combination of the single transformed representation, the input text, and a prefix text into the single combined representation;
decoding, via the text decoder, a suffix text from the single combined representation; and
concatenating the prefix text and the suffix text into the decoded output text.
13 . The system of claim 9 , wherein the operations further comprise:
projecting, via a fully connected layer, the single transformed representation to a same dimension with the pretrained language model before feeding the single transformed representation to the pretrained language model.
14 . The system of claim 9 , wherein the decoded output text is generated token by token by the pretrained language model conditioned on previously generated tokens.
15 . The system of claim 9 , wherein the query transformer includes a fewer number of parameters compared to the image encoder or the pretrained language model.
16 . The system of claim 9 , wherein the query transformer is trained with a training dataset of images and accompanying texts while the image encoder and the pretrained language model are kept frozen during training, and
wherein the training dataset of images and accompanying texts does not correspond to a particular vision-language task.
17 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for a multimodal framework for vision-language tasks, the instructions being executed by one or more processors to perform operations comprising:
receiving, at a user interface and at the multimodal framework comprising an image encoder, a query transformer, and a pretrained language model, an input image and an input utterance instructing a task relating to the input image;
encoding, by the image encoder, the input image into an image representation;
transforming, by the query transformer connected to an output of the image encoder, the image representation and a set of query vectors to extract visual features from the image representation into a single transformed representation, wherein the query transformer comprises a self-attention layer to generate a self-attention output from the set of query vectors and a cross-attention layer connected to an output of the self-attention layer and the output of the image encoder to generate a cross-attention output from the image representation and the self-attention output;
generating, by the pretrained language model, a single combined representation from a first input of the single transformed representation from the query transformer and a second input of the input utterance;
generating, by the pretrained language model, a decoded output text based on the single combined representation; and
presenting, via the user interface, a response based on the decoded output text in response to the input utterance.
18 . The non-transitory processor-readable storage medium of claim 17 , wherein the input utterance indicates an expected output text to generate based on visual content of the input image.
19 . The non-transitory processor-readable storage medium of claim 17 , wherein the pretrained language model includes a text decoder, and wherein the text decoder receives an input which combines the single transformed representation and tokens of the input utterance.
20 . The non-transitory processor-readable storage medium of claim 17 , wherein the query transformer is trained with a training dataset of images and accompanying texts while the image encoder and the pretrained language model are kept frozen during training, and}
wherein the training dataset of images and accompanying texts does not correspond to a particular vision-language task.