Multi-modal prompt learning for representation transfer on image recognition tasks
A method and system for multi-modal prompt learning of vision-language models. Encodings of image-text pairs can be combined with image prompts and text prompts before being input into an image encoder and text encoder of a vision-language model respectively. The image prompt can be generated using the text prompt using a vision-language coupling function to encourage synergy between the two prompts. The combination of encodings and prompts can be fed through the transformer layers of the encoders, and the output of each layer can be combined with a new prompt before entering the next layer, up until a specific depth. The subsequent transformer layers can process the output and generate a final representation for the image and text which can then be used for downstream tasks.
1 . A method for using a neural network to perform an image recognition task, comprising:
obtaining a pretrained vision-language model that includes a language branch and a vision branch, the vision-language model being a neural network that was pretrained using a web-scale training dataset;
simultaneously learning prompts for both the language and vision branches of the vision-language model, so as to fine tune the vision-language model to the image recognition task; and
applying a task input to the fine tuned vision-language model to perform the image recognition task on the task input,
wherein the step of simultaneously learning the prompts further comprises:
receiving an image-text pair and a ground truth value corresponding to the image-text pair,
initializing a plurality of learnable text prompt tokens,
inputting a text of the image-text pair into a text encoder included in the language branch, wherein the text is tokenized and projected to a plurality of word embeddings, and wherein the text encoder outputs a final text representation based on the plurality of word embeddings and the plurality of learnable text prompt tokens,
generating a plurality of learnable image prompt tokens using the plurality of learnable text prompt tokens,
inputting an image of the image-text pair into an image encoder included in the vision branch, wherein the image is split into a plurality of patch embeddings, wherein a learnable class token is appended to the plurality of patch embeddings, and wherein the image encoder outputs a final image representation based on the plurality of patch embeddings, the learnable class token, and the plurality of learnable image prompt tokens,
using the final text representation and the final image representation to perform image classification to generate a prediction, and
based on a comparison between the generated prediction and the ground truth value, updating the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token,
wherein the text encoder comprises K total number of text transformer layers, and wherein the image encoder comprises K total number of image transformer layers, wherein, at each text transformer layer up to a specific depth less than the total number of image transformer layers, the text encoder concatenates the plurality of word embeddings received as input to the text transformer layer with at least one of the plurality of learnable text prompt tokens, and wherein, at each image transformer layer up until the specific depth, the image encoder concatenates the plurality of patch embeddings received as input to the image transformer layer with at least one of the plurality of learnable image prompt tokens, and
wherein during the step of simultaneously learning the prompts, only the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token are updated, while other parts of the vision-language model are kept frozen.
2 . The method of claim 1 , wherein the text encoder forms the language branch of the vision-language model, and wherein the image encoder forms the vision branch of the vision-language model.
3 . The method of claim 2 , wherein the vision-language model comprises a contrastive language-image pre-training (CLIP) neural network.
4 . The method of claim 1 , wherein generating the plurality of learnable image prompt tokens using the plurality of learnable text prompt tokens further comprises:
for each learnable image prompt token of the plurality of learnable image prompt tokens, inputting one corresponding learnable text prompt token of the plurality of learnable text prompt tokens into a vision-language projection coupling function to generate the learnable image prompt token.
5 . The method of claim 4 , wherein the vision-language projection coupling function comprises a linear layer that maps inputs of a first dimensionality to outputs of a second dimensionality.
6 . The method of claim 1 , wherein the generated prediction includes a computation of a cosine similarity score.
7 . The method of claim 6 , further comprising:
computing a loss between the generated prediction and the ground truth value using a loss function;
computing a gradient of the loss function; and
backpropagating the gradient to update the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token.
8 . The method of claim 7 , wherein the loss function is a cross-entropy loss function or a Hinge loss function.
9 . The method of claim 1 , wherein the plurality of learnable text prompt tokens comprise natural language.
10 . The method of claim 1 , wherein the final text representation and the final image representation are stored in a representation table.
11 . The method of claim 1 , wherein the final text representation and the final image representation are elements of a common vision-language latent embedding space.
12 . The method of claim 1 , wherein the image classification comprises zero-shot classification.
13 . The method of claim 1 , wherein generating the plurality of learnable image prompt tokens using the plurality of learnable text prompt tokens further comprises:
for each learnable image prompt token of the plurality of learnable image prompt tokens, inputting one corresponding learnable text prompt token of the plurality of learnable text prompt tokens into a vision-language projection coupling function to generate the learnable image prompt token;
wherein the vision-language projection coupling function comprises a linear layer that maps inputs of a first dimensionality to outputs of a second dimensionality; and
wherein the downstream task is zero-shot classification, and wherein the generated prediction includes a computation of a cosine similarity score;
computing a loss between the generated prediction and the ground truth value using a cross-entropy loss function or a Hinge loss function;
computing a gradient of the loss function; and
backpropagating the gradient to update the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token.
14 . A non-transitory computer readable medium having instructions stored therein that, when executed by one or more processors, cause the one or more processors to perform a method for using a neural network to perform an image recognition task, the method comprising:
obtaining a pretrained vision-language model that includes a language branch and a vision branch, the vision-language model, the vision-language model being a neural network that was pretrained using a web-scale training dataset;
simultaneously learning prompts for both the language and vision branches of the vision-language model, so as to fine tune the vision-language model to the image recognition task; and
applying a task input to the fine tuned vision-language model to perform the image recognition task on the task input,
wherein the step of simultaneously learning the prompts further comprises:
receiving an image-text pair and a ground truth value corresponding to the image-text pair,
initializing a plurality of learnable text prompt tokens,
inputting a text of the image-text pair into a text encoder included in the language branch, wherein the text is tokenized and projected to a plurality of word embeddings, and wherein the text encoder outputs a final text representation based on the plurality of word embeddings and the plurality of learnable text prompt tokens,
generating a plurality of learnable image prompt tokens using the plurality of learnable text prompt tokens,
inputting an image of the image-text pair into an image encoder included in the vision branch, wherein the image is split into a plurality of patch embeddings, wherein a learnable class token is appended to the plurality of patch embeddings, and wherein the image encoder outputs a final image representation based on the plurality of patch embeddings, the learnable class token, and the plurality of learnable image prompt tokens,
using the final text representation and the final image representation to perform image classification to generate a prediction, and
based on a comparison between the generated prediction and the ground truth value, updating the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token,
wherein the text encoder comprises K total number of text transformer layers, and wherein the image encoder comprises K total number of image transformer layers, wherein, at each text transformer layer up to a specific depth J less than the total number of image transformer layers, the text encoder concatenates the plurality of word embeddings received as input to the text transformer layer with at least one of the plurality of learnable text prompt tokens, and wherein, at each image transformer layer up until the specific depth, the image encoder concatenates the plurality of patch embeddings received as input to the image transformer layer with at least one of the plurality of learnable image prompt tokens, and
wherein during the step of simultaneously learning the prompts, only the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token are updated, while other parts of the vision-language model are kept frozen.
15 . The non-transitory computer readable medium of claim 14 , wherein the neural network implements a contrastive language-image pre-training (CLIP) neural network, wherein the text encoder forms the language branch of the CLIP neural network, and wherein the image encoder forms the vision branch of the CLIP neural network.
16 . The non-transitory computer readable medium of claim 14 , wherein generating the plurality of learnable image prompt tokens using the plurality of learnable text prompt tokens comprises for each learnable image prompt token of the plurality of learnable image prompt tokens, inputting one corresponding learnable text prompt token of the plurality of learnable text prompt tokens into a vision-language projection coupling function to generate the learnable image prompt token, wherein the vision-language projection coupling function comprises a linear layer that maps inputs of a first dimensionality to outputs of a second dimensionality.
17 . The non-transitory computer readable medium of claim 14 , wherein the generated prediction includes a computation of a cosine similarity score computing a loss between the generated prediction and the ground truth value using a loss function, the method further comprising:
computing a gradient of the loss function; and
backpropagating the gradient to update the plurality of learnable text prompt tokens, the plurality of learnable image prompt tokens, and the learnable class token.
18 . The non-transitory computer readable medium of claim 14 , wherein the plurality of learnable text prompt tokens comprise natural language.