Pre-training a model using unlabeled videos

1. A method for training a machine learning model, the method comprising:

receiving unlabeled multimedia data, the multimedia data including a plurality of video frames and a plurality of transcribed utterances;

extracting one or more clips of multimedia data, the one or more clips including a subset of video frames of the plurality of video frames and at least a first utterance and a second utterance of the plurality of transcribed utterances associated with the subset of video frames, wherein the first utterance and second utterance occur at different times within the subset of video frames; and

training the machine learning model using the one or more clips of multimedia data, wherein the machine learning model comprises an encoder model and a decoder model, wherein training the machine learning model comprises:

inputting the subset of video frames and the first utterance into the machine learning model;

using the decoder model to generate a predicted utterance based on the subset of video frames and the first utterance; and

jointly updating one or more parameters of the encoder model and the decoder model based on a loss function that compares the predicted utterance with the second utterance.

2. The method of claim 1 , wherein training the machine learning model comprises:

performing forward generation on each of the one or more clips of multimedia data to train the machine-learning model, wherein, in said forward generation, the second utterance is temporally subsequent to the first utterance.

3. The method of claim 2 , wherein performing forward generation further comprises minimizing the negative log-likelihood of the predicted utterance with respect to the second utterance.

4. The method of claim 1 , wherein training the machine learning model comprises:

performing backward generation on each of the one or more clips of multimedia data to train the machine-learning model, wherein, in said backward generation, the second utterance is temporally prior to the first utterance.

5. The method of claim 4 , wherein backward generation further comprises minimizing the negative log-likelihood of the predicted utterance with respect to the second utterance.

6. The method of claim 1 , wherein each of the first utterance and the second utterance is a textual transcription of a spoken sentence.

7. The method of claim 1 , wherein training the machine learning model further comprises:

masking at least a portion of the first utterance;

performing masked language modeling loss on the first utterance to obtain a masked loss; and

training the machine learning model using the masked loss.

8. The method of claim 7 , wherein the masked loss is applied to outputs of the decoder model of the machine-learning model.

9. The method of claim 7 , wherein the masked loss is applied to both forward generation and backward generation training of the machine learning model.

10. The method of claim 1 , further comprising fine-tuning the machine learning model to receive unlabeled multimedia data and output one or more captions for video frames of the unlabeled multimedia data.

11. A system for training a machine learning model, the system comprising:

one or more processors; and

a memory comprising the machine learning model and one or more instructions that, when executed by the one or more processors, cause the one or more processors to perform a process, the process comprising:

receiving unlabeled multimedia data, the multimedia data including a plurality of video frames and a plurality of transcribed utterances;

extracting one or more clips of multimedia data, the one or more clips including a subset of video frames of the plurality of video frames and at least a first utterance and a second utterance of the plurality of transcribed utterances associated with the subset of video frames, wherein the first utterance and second utterance occur at different times within the subset of video frames; and

training the machine learning model using the one or more clips of multimedia data, wherein the machine learning model comprises an encoder model and a decoder model, wherein training the machine learning model comprises:

inputting the subset of video frames and the first utterance into the machine learning model;

using the decoder model to generate a predicted utterance based on the subset of video frames and the first utterance; and

jointly updating one or more parameters of the encoder model and the decoder model based on a loss function that compares the predicted utterance with the second utterance.

12. The system of claim 11 , wherein training the machine learning model comprises:

performing forward generation on each of the one or more clips of multimedia data to train the machine-learning model, wherein, in said forward generation, the second utterance is temporally subsequent to the first utterance.

13. The system of claim 12 , wherein performing forward generation further comprises minimizing the negative log-likelihood of the predicted utterance with respect to the second utterance.

14. The system of claim 11 , wherein training the machine learning model comprises:

performing backward generation on each of the one or more clips of multimedia data to train the machine-learning model, wherein, in said backward generation, the second utterance is temporally prior to the first utterance.

15. The system of claim 14 , wherein backward generation further comprises minimizing the negative log-likelihood of the predicted utterance with respect to the second utterance.

16. The system of claim 11 , wherein training the machine learning model further comprises:

masking at least a portion of the first utterance;

performing masked language modeling loss on the first utterance to obtain a masked loss; and

training the machine learning model using the masked loss.

17. The system of claim 16 , wherein the masked loss is applied to outputs of the decoder model of the machine-learning model.

18. The system of claim 16 , wherein the masked loss is applied to both forward generation and backward generation training of the machine learning model.

19. The system of claim 11 , further comprising fine-tuning the machine learning model to receive unlabeled multimedia data and output one or more captions for video frames of the unlabeled multimedia data.

20. A method for performing captioning for image or video data, the method comprising:

receiving unlabeled multimedia data, the multimedia data including a plurality of video frames and a plurality of transcribed utterances; and

outputting, from a machine learning model, one or more captions for the multimedia data based on the received multimedia data, wherein the machine learning model has been previously trained by performing one or more operations, the one or more operations comprising:

receiving unlabeled multimedia data, the multimedia data including a plurality of video frames and a plurality of transcribed utterances;

extracting one or more clips of multimedia data, the one or more clips including a subset of video frames of the plurality of video frames and at least a first utterance and a second utterance of the plurality of transcribed utterances associated with the subset of video frames, wherein the first utterance and second utterance occur at different times within the subset of video frames; and

training the machine learning model using the one or more clips of multimedia data, wherein the machine learning model comprises an encoder model and a decoder model, wherein training the machine learning model comprises:

inputting the subset of video frames and the first utterance into the machine learning model;

using the decoder model to generate a predicted utterance based on the subset of video frames and the first utterance; and

jointly updating one or more parameters of the encoder model and the decoder model based on a loss function that compares the predicted utterance with the second utterance.