System and method for supervised contrastive learning for multi-modal tasks

1. A method comprising:

obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and

training a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:

a first multi-modal representation loss based on the paired image-text pair; and

a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image.

2. The method of claim 1 , further comprising:

for each paired image-text pair, determining the second multi-modal representation loss by calculating a first loss associated with the first unpaired image-text pair and a second loss associated with the second unpaired image-text pair.

3. The method of claim 1 , wherein the losses further include, for each paired image-text pair:

a third multi-modal representation loss based on the paired image-text pair in which a representation of a first object in the image of the paired image-text pair is replaced with a representation of a second object that is of a same type as the first object; and

a fourth multi-modal representation loss based on the paired image-text pair in which a representation of the first object in the image of the paired image-text pair is replaced with a representation of a third object that is of a different type than the first object.

4. The method of claim 1 , wherein, in each paired image-text pair, at least one tag of the image matches at least one tag of the text.

5. The method of claim 1 , wherein the machine learning model is trained in association with a downstream task comprising at least one of: visual question answering (VQA), graph question answering (GQA), referring expressions, caption generation, and image retrieval.

6. The method of claim 1 , wherein the first multi-modal representation loss comprises a cross-entropy loss or an object detection loss.

7. A method comprising:

obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and

training a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:

a first multi-modal representation loss based on the paired image-text pair; and

a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair;

wherein training the machine learning model comprises:

determining word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs;

determining visual features for the images of the paired image-text pairs and the unpaired image-text pairs; and

implementing a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs.

8. An electronic device comprising:

at least one memory configured to store instructions; and

at least one processing device configured when executing the instructions to:

obtain a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and

train a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:

a first multi-modal representation loss based on the paired image-text pair; and

a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image.

9. The electronic device of claim 8 , wherein the at least one processing device is further configured when executing the instructions to:

for each paired image-text pair, determine the second multi-modal representation loss by calculating a first loss associated with the first unpaired image-text pair and a second loss associated with the second unpaired image-text pair.

10. The electronic device of claim 8 , wherein the losses further include, for each paired image-text pair:

a third multi-modal representation loss based on the paired image-text pair in which a representation of a first object in the image of the paired image-text pair is replaced with a representation of a second object that is of a same type as the first object; and

a fourth multi-modal representation loss based on the paired image-text pair in which a representation of the first object in the image of the paired image-text pair is replaced with a representation of a third object that is of a different type than the first object.

11. The electronic device of claim 8 , wherein, in each paired image-text pair, at least one tag of the image matches at least one tag of the text.

12. The electronic device of claim 8 , wherein, to train the machine learning model, the at least one processing device is configured when executing the instructions to:

determine word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs;

determine visual features for the images of the paired image-text pairs and the unpaired image-text pairs; and

implement a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs.

13. The electronic device of claim 8 , wherein the at least one processing device is configured when executing the instructions to train the machine learning model in association with a downstream task comprising at least one of: visual question answering (VQA), graph question answering (GQA), referring expressions, caption generation, and image retrieval.

14. The electronic device of claim 8 , wherein the first multi-modal representation loss comprises a cross-entropy loss or an object detection loss.

15. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

obtain a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and

train a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:

a first multi-modal representation loss based on the paired image-text pair; and

a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image.

16. The non-transitory machine-readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:

for each paired image-text pair, determine the second multi-modal representation loss by calculating a first loss associated with the first unpaired image-text pair and a second loss associated with the second unpaired image-text pair.

17. The non-transitory machine-readable medium of claim 15 , wherein the losses further include, for each paired image-text pair:

a third multi-modal representation loss based on the paired image-text pair in which a representation of a first object in the image of the paired image-text pair is replaced with a representation of a second object that is of a same type as the first object; and

a fourth multi-modal representation loss based on the paired image-text pair in which a representation of the first object in the image of the paired image-text pair is replaced with a representation of a third object that is of a different type than the first object.

18. The non-transitory machine-readable medium of claim 15 , wherein, in each paired image-text pair, at least one tag of the image matches at least one tag of the text.

19. The non-transitory machine-readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to train the machine learning model comprise instructions that when executed cause the at least one processor to:

determine word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs;

determine visual features for the images of the paired image-text pairs and the unpaired image-text pairs; and

implement a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs.

20. The non-transitory machine-readable medium of claim 15 , wherein the first multi-modal representation loss comprises a cross-entropy loss or an object detection loss.