Systems and methods for a vision-language pretraining framework
Embodiments described herein provide a multimodal vision-language model. The multimodal vision-language model contains a Generalist Multimodal Transformer capable of complete multiple tasks using the same set of parameters learning from pre-training. The Generalist Multimodal Transformer allows alignment between frozen, unimodal encoders, such as image encoders and large language models. The Generalist Multimodal Transformer eliminates the need for fine-tuning the image encoders and large language models.
1 . A method for pre-training a multimodal framework for vision-language tasks, the method comprising:
receiving, from a communication interface, an image and a text accompanying the image;
encoding, by an image encoder, the image into an image representation;
transforming, by a query transformer, the image representation and a set of queries into a transformed representation;
generating, by the query transformer, a text representation based at least in part from the text;
training the query transformer according to one or more vision-language training objectives computed based on the transformed representation and the text representation while keeping the image encoder frozen;
generating, by a pretrained language model, a decoded output text based on an output representation from the updated query transformer;
computing a loss based on the decoded output text and the text accompanying the image; and
training the query transformer based on the loss while keeping the image encoder and the pretrained language model frozen, wherein the pretrained language model includes a text decoder, and wherein the generating, by the pretrained language model, the decoded output text based on the output representation from the updated query transformer comprises:
projecting, via a fully connected layer, the output representation to a same dimension with the pretrained language model; and
generating, by the text decoder, the decoded output text based on the projected output representation.
2 . The method of claim 1 , wherein the set of queries are a set of learnable embeddings, and wherein the transforming, by the query transformer, the image representation and the set of queries into the transformed representation includes:
generating, at one or more attention layers of an image transformer in the query transformer, query embeddings from the set of queries.
3 . The method of claim 2 , wherein the one or more vision-language training objectives comprises an image-text matching objective that is generated by:
applying a self-attention mask between the set of queries and the text to generate the query embeddings;
generating, via a classifier head, a match prediction indicating whether the image and the text are a matching pair based on the query embeddings; and
computing the image-text matching objective based on the match prediction and a ground truth.
4 . The method of claim 2 , wherein the one or more vision-language training objectives comprises an image-text contrastive learning objective that is generated by:
computing an image-text similarity based on the query embeddings and the text representation; and
computing the image-text contrastive learning objective based on the image-text similarity.
5 . The method of claim 2 , wherein the one or more vision-language training objectives comprises an image-grounded text generation objective that is generated by:
applying a multi-modal self-attention mask to the set of queries and the text;
generating a predicted text conditioned on image features based on the applied multi-modal self-attention mask; and
computing the image-grounded text generation objective based on the predicted text and the text.
6 . The method of claim 1 , wherein the training the query transformer according to one or more vision-language training objectives comprises:
updating parameters of the query transformer via backpropagation based on any joint combination of the one or more vision-language training objectives.
7 . The system of claim 1 , wherein the decoded output text is generated token by token by the pretrained language model conditioned on previously generated tokens.
8 . The method of claim 1 , wherein the pretrained language model includes a text encoder and a text decoder, and wherein the generating, by the pretrained language model, the decoded output text based on the output representation from the updated query transformer comprises:
projecting, via a fully connected layer, the output representation to a same dimension with the pretrained language model;
encoding, via the text encoder, the projected output representation prepended to a prefix text into a prefix representation;
decoding, via the text decoder, a suffix text from the prefix representation; and
concatenating the prefix text and the suffix text into the decoded output text.
9 . The method of claim 1 , wherein the decoded output text is generated token by token by the pretrained language model conditioned on previously generated tokens.
10 . The method of claim 1 , wherein the query transformer is first updated according to the one or more vision-language training objectives, and then updated based on the loss.
11 . The system of claim 1 , wherein the pretrained language model includes a text encoder and a text decoder, and wherein the operation of generating, by the pretrained language model, the decoded output text based on the output representation from the updated query transformer comprises:
projecting, via a fully connected layer, the output representation to a same dimension with the pretrained language model;
encoding, via the text encoder, the projected output representation prepended to a prefix text into a prefix representation;
decoding, via the text decoder, a suffix text from the prefix representation; and
concatenating the prefix text and the suffix text into the decoded output text.
12 . A system for pre-training a multimodal framework for vision-language tasks, the system comprising:
a communication interface receiving an image and a text accompanying the image;
a memory storing an image encoder, a query transformer, a pretrained language model, and a plurality of processor-executable instructions; and
one or more processors executing the instructions to perform operations including:
encoding, by the image encoder, the image into an image representation;
transforming, by the query transformer, the image representation and a set of queries into a transformed representation;
generating, by the query transformer, a text representation based at least in part from the text;
training the query transformer according to one or more vision-language training objectives computed based on the transformed representation and the text representation while keeping the image encoder frozen;
generating, by the pretrained language model, a decoded output text based on an output representation from the updated query transformer;
computing a loss based on the decoded output text and the text accompanying the image; and
training the query transformer based on the loss while keeping the image encoder and the pretrained language model frozen, wherein the pretrained language model includes a text decoder, and wherein the generating, by the pretrained language model, the decoded output text based on the output representation from the updated query transformer comprises:
projecting, via a fully connected layer, the output representation to a same dimension with the pretrained language model; and
generating, by the text decoder, the decoded output text based on the projected output representation.
13 . The system of claim 12 , wherein the set of queries are a set of learnable embeddings, and wherein the transforming, by the query transformer, the image representation and the set of queries into the transformed representation includes:
generating, at one or more attention layers of an image transformer in the query transformer, query embeddings from the set of queries.
14 . The system of claim 13 , wherein the one or more vision-language training objectives comprises an image-text contrastive learning objective that is generated by:
computing an image-text similarity based on the query embeddings and the text representation; and
computing the image-text contrastive learning objective based on the image-text similarity.
15 . The system of claim 13 , wherein the one or more vision-language training objectives comprises an image-grounded text generation objective that is generated by:
applying a multi-modal self-attention mask to the set of queries and the text;
generating a predicted text conditioned on image features based on the applied multi-modal self-attention mask; and
computing the image-grounded text generation objective based on the predicted text and the text.
16 . The system of claim 13 , wherein the one or more vision-language training objectives comprises an image-text matching objective that is generated by:
applying a self-attention mask between the set of queries and the text to generate the query embeddings;
generating, via a classifier head, a match prediction indicating whether the image and the text are a matching pair based on the query embeddings; and
computing the image-text matching objective based on the match prediction and a ground truth.
17 . The system of claim 12 , wherein the operation of training the query transformer according to one or more vision-language training objectives comprises:
updating parameters of the query transformer via backpropagation based on any joint combination of the one or more vision-language training objectives.
18 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for pre-training a multimodal framework for vision-language tasks, the instructions executed by one or more processors to perform operations, the method comprising:
receiving, from a communication interface, an image and a text accompanying the image;
encoding, by an image encoder, the image into an image representation;
transforming, by a query transformer, the image representation and a set of queries into a transformed representation;
generating, by the query transformer, a text representation based at least in part from the text;
training the query transformer according to one or more vision-language training objectives computed based on the transformed representation and the text representation while keeping the image encoder frozen;
generating, by a pretrained language model, a decoded output text based on an output representation from the updated query transformer;
computing a loss based on the decoded output text and the text accompanying the image; and
training the query transformer based on the loss while keeping the image encoder and the pretrained language model frozen, wherein the pretrained language model includes a text decoder, and wherein the generating, by the pretrained language model, the decoded output text based on the output representation from the updated query transformer comprises:
projecting, via a fully connected layer, the output representation to a same dimension with the pretrained language model; and
generating, by the text decoder, the decoded output text based on the projected output representation.
19 . The non-transitory processor-readable storage medium of claim 18 , wherein the set of queries are a set of learnable embeddings, and wherein the transforming, by the query transformer, the image representation and the set of queries into the transformed representation includes:
generating, at one or more attention layers of an image transformer in the query transformer, query embeddings from the set of queries.
20 . The non-transitory processor-readable storage medium of claim 19 , wherein the one or more vision-language training objectives comprises an image-text matching objective that is generated by:
applying a self-attention mask between the set of queries and the text to generate the query embeddings;
generating, via a classifier head, a match prediction indicating whether the image and the text are a matching pair based on the query embeddings; and
computing the image-text matching objective based on the match prediction and a ground truth.