Systems and methods for unified vision-language understanding and generation
Embodiments described herein provide systems, methods, and devices for generating enhanced vison-language training data. A method may include: receiving, from a communication interface, a first training dataset of image-text pairs and a second training dataset of annotated image-text pairs; fine-tuning an image-grounded text decoder and an image-grounded text encoder using the second training dataset of annotated image-text pairs; generating, by the fine-tuned image-grounded text decoder, a predicted text based on a training image from the first training dataset; generating, by the fine-tuned image-grounded text encoder, a filtering decision based on the training image and the predicted text; adding the training image and the predicted text to form a third training dataset of image-text pairs depending on the filter decision; and training a vision-language model using the third training dataset of image-text pairs.
1. A method of generating enhanced vison-language training data, the method comprising:
receiving, from a communication interface, a first training dataset of image-text pairs and a second training dataset of annotated image-text pairs;
fine-tuning an image-grounded text decoder and an image-grounded text encoder using the second training dataset of annotated image-text pairs;
generating, by the fine-tuned image-grounded text decoder, a predicted text based on a training image from the first training dataset;
generating, by the fine-tuned image-grounded text encoder, a filtering decision based on the training image and the predicted text;
adding the training image and the predicted text to form a third training dataset of image-text pairs depending on the filter decision; and
training a vision-language model using the third training dataset of image-text pairs.
2. The method of claim 1 , wherein the image-grounded text decoder is finetuned by:
generating a predicted text in response to an image in the second training dataset; and
computing a language modeling loss comparing the predicted text with an annotated text paired with the image.
3. The method of claim 1 , wherein the image-grounded text encoder is finetuned by:
generating a text encoding of a text from the second training dataset;
generating an image encoding of an image pairing the text from the second training dataset; and
computing an image-text contrastive loss based on a positive pair of the text encoding and the image encoding, and negative pairs of the image encoding paired with other text encodings.
4. The method of claim 1 , wherein the image-grounded text encoder is finetuned by:
generating a binary classification indicating whether a text and an image from the second training dataset are a match; and
computing an image-text matching loss comparing the binary classification and a ground truth.
5. The method of claim 1 , wherein the filter decision is generated by generating a binary classification indicating whether an input image and an input text matches.
6. The method of claim 5 , wherein the third training dataset is formed by adding the input image and the input text as a training pair when the binary classification indicates the input image and the input text match.
7. The method of claim 1 , further comprising:
adding the second training dataset to the third training dataset.
8. The method of claim 1 , wherein the vision-language model includes any combination of an image encoder, a text encoder, an image-grounded text encoder or an image grounded text decoder.
9. The method of claim 1 , wherein the image-grounded text decoder and the image-grounded text encoder at least partially share parameters.
10. A system of generating enhanced vison-language training data, the system comprising:
a communication interface that receives a first training dataset of image-text pairs and a second training dataset of annotated image-text pairs;
a memory storing a plurality of processor-executable instructions; and
a processor executing the plurality of processor-executable instructions to perform operations comprising:
fine-tuning an image-grounded text decoder and an image-grounded text encoder using the second training dataset of annotated image-text pairs;
generating, by the fine-tuned image-grounded text decoder, a predicted text based on a training image from the first training dataset;
generating, by the fine-tuned image-grounded text encoder, a filtering decision based on the training image and the predicted text;
adding the training image and the predicted text to form a third training dataset of image-text pairs depending on the filter decision; and
training a vision-language model using the third training dataset of image-text pairs.
11. The system of claim 10 , wherein the image-grounded text decoder is finetuned by:
generating a predicted text in response to an image in the second training dataset; and
computing a language modeling loss comparing the predicted text with an annotated text paired with the image.
12. The system of claim 10 , wherein the image-grounded text encoder is finetuned by:
generating a text encoding of a text from the second training dataset;
generating an image encoding of an image pairing the text from the second training dataset; and
computing an image-text contrastive loss based on a positive pair of the text encoding and the image encoding, and negative pairs of the image encoding paired with other text encodings.
13. The system of claim 10 , wherein the image-grounded text encoder is finetuned by:
generating a binary classification indicating whether a text and an image from the second training dataset are a match; and
computing an image-text matching loss comparing the binary classification and a ground truth.
14. The system of claim 10 , wherein the filter decision is generated by generating a binary classification indicating whether an input image and an input text matches.
15. The system of claim 14 , wherein the third training dataset is formed by adding the input image and the input text as a training pair when the binary classification indicates the input image and the input text match.
16. The system of claim 10 , wherein the operations further comprise:
adding the second training dataset to the third training dataset.
17. The system of claim 10 , wherein the vision-language model includes any combination of an image encoder, a text encoder, an image-grounded text encoder or an image grounded text decoder.
18. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for generating enhanced vison-language training data, the instructions being executed by a processor to perform operations comprising:
receiving, from a communication interface, a first training dataset of image-text pairs and a second training dataset of annotated image-text pairs;
fine-tuning an image-grounded text decoder and an image-grounded text encoder using the second training dataset of annotated image-text pairs;
generating, by the fine-tuned image-grounded text decoder, a predicted text based on a training image from the first training dataset;
generating, by the fine-tuned image-grounded text encoder, a filtering decision based on the training image and the predicted text;
adding the training image and the predicted text to form a third training dataset of image-text pairs depending on the filter decision; and
training a vision-language model using the third training dataset of image-text pairs.
19. The non-transitory processor-readable storage medium of claim 18 , wherein the image-grounded text encoder is finetuned by:
generating a binary classification indicating whether a text and an image from the second training dataset are a match; and
computing an image-text matching loss comparing the binary classification and a ground truth.
20. The non-transitory processor-readable storage medium of claim 18 , wherein the vision-language model includes any combination of an image encoder, a text encoder, an image-grounded text encoder or an image grounded text decoder.