Systems and methods for unified vision-language understanding and generation
Embodiments described herein provide systems, methods, and devices for pre-training a multimodal encoder-decoder (MED) model for vision-language tasks. A method may include encoding, by an image encoder of the MED, an image into an image representation; encoding, by a text encoder of the MED, a text into a text representation; generating, by an image-grounded text encoder of the MED, a multimodal representation based on the image representation and the text; generating, by an image-grounded text decoder of the MED, a predicted text based on the image representation and the text; generating, through an image-text matching (ITM) head, a binary classification indicating whether the image and the text are a match; computing a first loss, ITM loss, and third loss based on the image representation, text representation, binary classification, predicted text and text; jointly updating the MED based on the first loss, the second loss and the third loss.
1. A method for pre-training a multimodal encoder-decoder (MED) model for vision-language tasks, the method comprising:
receiving, from a communication interface, an image and a text accompanying the image;
encoding, by an image encoder of the MED, the image into an image representation;
encoding, by a text encoder of the MED, the text into a text representation;
generating, by an image-grounded text encoder of the MED, a multimodal representation based on the image representation and the text;
generating, by an image-grounded text decoder of the MED, a predicted text based on the image representation and the text;
generating, through an image-text matching (ITM) head, a binary classification indicating whether the image and the text are a match based on the multimodal representation;
computing a first loss based on the image representation and the text representation;
computing a second loss based on the binary classification, wherein the second loss is an ITM loss;
computing a third loss based on the predicted text and the text; and
jointly updating the MED based on the first loss, the second loss and the third loss, wherein the image-grounded text encoder of the MED is updated based on the second loss.
2. The method of claim 1 , further comprising operating the MED as the text encoder, the image-grounded text encoder or the image-grounded text decoder independently,
wherein the text encoder, the image-grounded text encoder or the image-grounded text decoder at least partially share parameters.
3. The method of claim 1 , wherein the encoding, by the image encoder of the MED comprises:
applying self-attention to the image; and
applying a feedforward layer to a self-attention output.
4. The method of claim 1 , wherein the encoding, by the text encoder of the MED, further comprises:
applying bidirectional self-attention to the text appended with a start token.
5. The method of claim 1 , wherein the generating, by an image-grounded text encoder of the MED, further comprises:
applying bidirectional self-attention to the text appended with an encoding indicator; and
applying cross-attention to a bidirectional self-attention output and the image representation.
6. The method of claim 1 , wherein the generating by the image-grounded text decoder of the MED, further comprises:
applying causal self-attention to the text appended with a decoding indicator; and
applying cross-attention to a causal self-attention output and the image representation.
7. The method of claim 1 , wherein the first loss is an image-text contrastive (ITC) loss computed from a positive pair of the image representation and the text representation, and a plurality of negative pairs of the image representation and negative text representations generated from texts that do not match with the image, and
wherein the first loss is used to update the image encoder and the text encoder.
8. The method of claim 1 , wherein the third loss is a language modeling loss computed as a cross-entropy between the predicted text and the text, and wherein the third loss is used to update the image-grounded text decoder.
9. The method of claim 1 , further comprising:
computing a weighted sum of the first loss, the second loss and the third loss; and
jointly updating the MED comprising the image encoder, the text encoder, the image-grounded text encoder and the image-grounded text decoder based on the weighted sum.
10. A system for pre-training a multimodal encoder-decoder (MED) model for vision-language tasks, the system comprising:
a communication interface that receives an image and a text accompanying the image;
a memory storing the MED model, and a plurality of processor-executable instructions; and
a processor executing the plurality of processor-executable instructions to perform operations comprising:
encoding, by an image encoder of the MED, the image into an image representation;
encoding, by a text encoder of the MED, the text into a text representation;
generating, by an image-grounded text encoder of the MED, a multimodal representation based on the image representation and the text;
generating, by an image-grounded text decoder of the MED, a predicted text based on the image representation and the text;
generating, through an image-text matching (ITM) head, a binary classification indicating whether the image and the text are a match based on the multimodal representation;
computing a first loss based on the image representation and the text representation;
computing a second loss based on the binary classification, wherein the second loss is an ITM loss;
computing a third loss based on the predicted text and the text; and
jointly updating the MED based on the first loss, the second loss and the third loss, wherein the image-grounded text encoder of the MED is updated based on the second loss.
11. The system of claim 10 , wherein the operations further comprise operating the MED as the text encoder, the image-grounded text encoder or the image-grounded text decoder independently,
wherein the text encoder, the image-grounded text encoder or the image-grounded text decoder at least partially share parameters.
12. The system of claim 10 , wherein an operation of encoding, by the image encoder of the MED comprises:
applying self-attention to the image; and
applying a feedforward layer to a self-attention output.
13. The system of claim 10 , wherein an operation of encoding, by the text encoder of the MED, further comprises:
applying bidirectional self-attention to the text appended with a start token.
14. The system of claim 10 , wherein an operation of generating, by an image-grounded text encoder of the MED, further comprises:
applying bidirectional self-attention to the text appended with an encoding indicator; and
applying cross-attention to a bidirectional self-attention output and the image representation.
15. The system of claim 10 , wherein an operation of generating by the image-grounded text decoder of the MED, further comprises:
applying causal self-attention to the text appended with a decoding indicator; and
applying cross-attention to a causal self-attention output and the image representation.
16. The system of claim 10 , wherein the first loss is an image-text contrastive (ITC) loss computed from a positive pair of the image representation and the text representation, and a plurality of negative pairs of the image representation and negative text representations generated from texts that do not match with the image, and
wherein the first loss is used to update the image encoder and the text encoder.
17. The system of claim 10 , wherein the third loss is a language modeling loss computed as a cross-entropy between the predicted text and the text, and wherein the third loss is used to update the image-grounded text decoder.
18. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for pre-training a multimodal encoder-decoder (MED) model for vision-language tasks, the instructions being executed by a processor to perform operations comprising:
receiving, from a communication interface, an image and a text accompanying the image;
encoding, by an image encoder of the MED, the image into an image representation;
encoding, by a text encoder of the MED, the text into a text representation;
generating, by an image-grounded text encoder of the MED, a multimodal representation based on the image representation and the text;
generating, by an image-grounded text decoder of the MED, a predicted text based on the image representation and the text;
generating, through an image-text matching (ITM) head, a binary classification indicating whether the image and the text are a match based on the multimodal representation;
computing a first loss based on the image representation and the text representation;
computing a second loss based on the binary classification, wherein the second loss is an ITM loss;
computing a third loss based on the predicted text and the text; and
jointly updating the MED based on the first loss, the second loss and the third loss, wherein the image-grounded text encoder of the MED is updated based on the second loss.
19. The non-transitory processor-readable storage medium of claim 18 , the operations further comprising operating the MED as the text encoder, the image-grounded text encoder or the image-grounded text decoder independently,
wherein the text encoder, the image-grounded text encoder or the image-grounded text decoder at least partially share parameters.
20. The non-transitory processor-readable storage medium of claim 18 , wherein the encoding, by the image encoder of the MED comprises:
applying self-attention to the image; and
applying a feedforward layer to a self-attention output.