Systems and methods for vision-and-language representation learning
Embodiments described herein provide visual-and-language (V+L) systems and methods for learning vision and language representations. Specifically, a method may comprise receiving a training dataset comprising a plurality of image samples and a plurality of text samples; encoding the plurality of image samples into a plurality of encoded image samples and the plurality of text samples into a plurality of encoded text samples; computing a first loss objective based on the plurality of encoded image samples and the plurality of encoded text samples; encoding a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples; computing a second loss objective based on the plurality of encoded image-text samples; and updating the V+L model based at least in part on the first loss objective and the second loss objective.
1. A method, performed by one or more processors, for training a vision-and-language learning (V+L) model comprising an image encoder, a text encoder, and a multimodal encoder, the method comprising:
receiving, via a data interface, a training dataset comprising a plurality of image samples and a plurality of text samples, at least one image sample of the plurality of image samples corresponding to at least one text sample of the plurality of text samples;
encoding, by an image encoder implemented by the one or more processors, the plurality of image samples into a plurality of encoded image samples and, by a text encoder, the plurality of text samples into a plurality of encoded text samples;
computing a first loss objective based on the plurality of encoded image samples and the plurality of encoded text samples;
encoding, by a multimodal encoder implemented by the one or more processors, a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples;
choosing the first subset and the second subset based at least in part on mining negative image-text pairs through contrastive similarity of the encoded image-text samples;
computing a second loss objective based on the plurality of encoded image-text samples; and
updating by the one or more processors the V+L model based at least in part on the first loss objective and the second loss objective.
2. The method of claim 1 , wherein the first loss objective comprises an image-text contrastive (ITC) loss objective that is an average expected sum of a cross-entropy between computed softmax-normalized image-to-text similarity and labeled ground-truth image-to-text similarity and a cross-entropy between computed softmax-normalized text-to-image similarity and labeled ground-truth text-to-image similarity.
3. The method of claim 1 , wherein the second loss objective comprises an image-text matching (ITM) loss objective that is computed as a cross-entropy between a predicted two-class probability of a image-text pair and a ground-truth one-hot two-class possibility, and an MLM loss objective that is computed as a cross-entropy between a predicted possibility of one or more masked tokens in the encoded image-text samples and a ground truth identity of the one or more masked tokens in the encoded image-text samples.
4. The method of claim 1 , wherein the updating the V+L model comprises:
updating the image encoder and the text encoder based at least in part on the first loss objective; and
updating the multimodal encoder based at least in part on the second loss objective.
5. The method of claim 1 , further comprising:
forming a momentum model using momentum distillation (MoD);
generating a plurality of modeled image samples and a plurality of modeled text samples using the momentum model;
including the plurality of modeled image samples in the plurality of image samples; and
including the plurality of modeled text samples in the plurality of text samples.
6. The method of claim 1 , wherein the image encoder, the text encoder, and the multimodal encoder each comprises a transformer.
7. The method of claim 1 , further comprising:
fine-tuning the V+L model for a task selected from the group consisting of: an image-text retrieval task, an image-to-text retrieval (TR) task, a text-to-image retrieval (IR) task, a visual entailment (VE) task, a visual question answering (VQA) task, and a natural language for visual reasoning for real (NLVR) task.
8. A system for training a V+L model, comprising:
a non-transitory memory; and
one or more processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:
receiving, via a data interface, a training dataset comprising a plurality of image samples and a plurality of text samples, at least one image sample of the plurality of image samples corresponding to at least one text sample of the plurality of text samples;
encoding, by an image encoder, the plurality of image samples into a plurality of encoded image samples and, by a text encoder, the plurality of text samples into a plurality of encoded text samples;
computing a first loss objective based on the plurality of encoded image samples and the plurality of encoded text samples;
encoding, by a multimodal encoder, a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples;
choosing the first subset and the second subset based at least in part on mining negative image-text pairs through contrastive similarity of the encoded image-text samples;
computing a second loss objective based on the plurality of encoded image-text samples; and
updating the V+L model of the image encoder, the text encoder, and the multimodal encoder based at least in part on the first loss objective and the second loss objective.
9. The system of claim 8 , wherein the updating the V+L model comprises updating the image encoder and the text encoder based at least in part on the first loss objective and updating the multimodal encoder based at least in part on the second loss objective.
10. The system of claim 8 , wherein the operations further comprise:
forming a momentum model using momentum distillation (MoD);
generating a plurality of modeled image samples and a plurality of modeled text samples using the momentum model;
including the plurality of modeled image samples in the plurality of image samples; and
including the plurality of modeled text samples in the plurality of text samples.
11. The system of claim 8 , wherein the image encoder, the text encoder, and the multimodal encoder each comprise a transformer.
12. The system of claim 8 , wherein the operations further comprise fine-tuning the V+L model for a task selected from the group consisting of: an image-text retrieval task, an image-to-text retrieval (TR) task, a text-to-image retrieval (IR) task, a visual entailment (VE) task, a visual question answering (VQA) task, and a natural language for visual reasoning for real (NLVR) task.
13. A non-transitory, machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising:
receiving, via a data interface, a training dataset comprising a plurality of image samples and a plurality of text samples, at least one image sample of the plurality of image samples corresponding to at least one text sample of the plurality of text samples;
encoding, by an image encoder, the plurality of image samples into a plurality of encoded image samples and, by a text encoder, the plurality of text samples into a plurality of encoded text samples;
computing a first loss objective based on the plurality of encoded image samples and the plurality of encoded text samples;
encoding, by a multimodal encoder, a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples;
choosing the first subset and the second subset based at least in part on mining negative image-text pairs through contrastive similarity of the encoded image-text samples;
computing a second loss objective based on the plurality of encoded image-text samples; and
updating the image encoder, the text encoder, and the multimodal encoder based at least in part on the first loss objective and the second loss objective.
14. The non-transitory, machine-readable medium of claim 13 , wherein the updating the V+L model comprises updating the image encoder and the text encoder based at least in part on the first loss objective and updating the multimodal encoder based at least in part on the second loss objective.
15. The non-transitory, machine-readable medium of claim 13 , wherein the operations further comprise:
forming a momentum model using momentum distillation (MoD);
generating a plurality of modeled image samples and a plurality of modeled text samples using the momentum model;
including the plurality of modeled image samples in the plurality of image samples; and
including the plurality of modeled text samples in the plurality of text samples.
16. The non-transitory, machine-readable medium of claim 13 , wherein the image encoder, the text encoder, and the multimodal encoder each comprise a transformer.
17. The non-transitory, machine-readable medium of claim 13 , wherein the operations further comprise fine-tuning the V+L model for a task selected from the group consisting of: an image-text retrieval task, an image-to-text retrieval (TR) task, a text-to-image retrieval (IR) task, a visual entailment (VE) task, a visual question answering (VQA) task, and a natural language for visual reasoning for real (NLVR) task.