Systems and methods for video and language pre-training

1. A method of video-and-language alignment contrastive pretraining, the method comprising:

obtaining, via a data interface, a plurality of video frames and a plurality of text descriptions corresponding to the plurality of video frames;

encoding, by a video encoder, the plurality of video frames into video feature representations;

encoding, by a text encoder, the plurality of text descriptions into text feature representations;

computing similarity scores between the video feature representations and the text feature representations;

computing a first contrastive loss based at least in part on an exponential of computed similarity scores corresponding to a matching pair of a first video frame and a first text description and one or more pairs of the first video frame and text descriptions that do not match the first video frame;

computing a second contrastive loss based at least in part on an exponential of computed similarity scores corresponding to the matching pair of the first video frame and the first text description and one or more pairs of the first text description and video frames that do not match the first text description;

computing a video-text contrastive loss by taking a weighted sum of the first and the second contrastive losses; and

updating the video encoder and the text encoder based at least in part on the video-text contrastive loss.

2. The method of claim 1 , wherein the plurality of video frames are sparsely sampled from a video stream.

3. The method of claim 1 , further comprising:

partitioning each video frame into a number of non-overlapping patches;

feeding the number of non-overlapping patches to a linear projection layer to produce a sequence of patch tokens;

appending a video start token to the sequence of patch tokens.

4. The method of claim 1 , wherein encoding, by the video encoder, the plurality of video frames into video feature representations further comprises:

applying self-attention along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features; and

applying a temporal fusion layer to the per-frame features along the temporal dimension to aggregate per-frame features into video features.

5. The method of claim 1 , wherein the video feature representations comprise a sequence of visual embeddings including an embedding of a video start token.

6. The method of claim 1 , wherein the text feature representations comprise a sequence of text embeddings including an embedding of a text start token.

7. The method of claim 1 , wherein the similarity scores are computed by:

transforming an embedding of a video start token from the video encoder into a normalized video embedding;

transforming an embedding of a text start token from the text encoder into a normalized text embedding; and

computing a dot product of the normalized video embedding and the normalized text embedding.

8. The method of claim 1 , further comprising:

encoding, by a multi-modal video-text encoder, the video feature representations and the text feature representations into a set of multimodal embeddings; and

generating, by a classifier, an entity prediction from the set of multimodal embeddings.

9. A system of video-and-language alignment contrastive pretraining, the system comprising:

a data interface obtaining a plurality of video frames and a plurality of text descriptions corresponding to the plurality of video frames;

a memory storing a plurality of processor-executable instructions; and

a processor executing the instructions to perform operations comprising:

encoding, by a video encoder, the plurality of video frames into video feature representations;

encoding, by a text encoder, the plurality of text descriptions into text feature representations;

computing similarity scores between the video feature representations and the text feature representations;

computing a first contrastive loss based at least in part on an exponential of computed similarity scores corresponding to a matching pair of a first video frame and a first text description and one or more pairs of the first video frame and text descriptions that do not match the first video frame;

computing a second contrastive loss based at least in part on an exponential of computed similarity scores corresponding to the matching pair of the first video frame and the first text description and one or more pairs of the first text description and video frames that do not match the first text description;

computing a video-text contrastive loss by taking a weighted sum of the first and the second contrastive losses; and

updating the video encoder and the text encoder based at least in part on the video-text contrastive loss.

10. The system of claim 9 , wherein the plurality of video frames are sparsely sampled from a video stream.

11. The system of claim 9 , wherein the operations further comprise:

partitioning each video frame into a number of non-overlapping patches;

feeding the number of non-overlapping patches to a linear projection layer to produce a sequence of patch tokens;

appending a video start token to the sequence of patch tokens.

12. The system of claim 9 , wherein an operation of encoding, by the video encoder, the plurality of video frames into video feature representations further comprises:

applying self-attention along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features; and

applying a temporal fusion layer to the per-frame features along the temporal dimension to aggregate per-frame features into video features.

13. The system of claim 9 , wherein the video feature representations comprise a sequence of visual embeddings including an embedding of a video start token.

14. The system of claim 9 , wherein the text feature representations comprise a sequence of text embeddings including an embedding of a text start token.

15. The system of claim 9 , wherein the similarity scores are computed by:

transforming an embedding of a video start token from the video encoder into a normalized video embedding;

transforming an embedding of a text start token from the text encoder into a normalized text embedding; and

computing a dot product of the normalized video embedding and the normalized text embedding.

16. The system of claim 9 , wherein the operations further comprise:

encoding, by a multi-modal video-text encoder, the video feature representations and the text feature representations into a set of multimodal embeddings; and

generating, by a classifier, an entity prediction from the set of multimodal embeddings.

17. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for video-and-language alignment contrastive pretraining, the instructions being executed by a processor to perform operations comprising:

obtaining, via a data interface, a plurality of video frames and a plurality of text descriptions corresponding to the plurality of video frames;

encoding, by a video encoder, the plurality of video frames into video feature representations;

encoding, by a text encoder, the plurality of text descriptions into text feature representations;

computing similarity scores between the video feature representations and the text feature representations;

computing a first contrastive loss based at least in part on an exponential of computed similarity scores corresponding to a matching pair of a first video frame and a first text description and one or more pairs of the first video frame and text descriptions that do not match the first video frame;

computing a second contrastive loss based at least in part on an exponential of computed similarity scores corresponding to the matching pair of the first video frame and the first text description and one or more pairs of the first text description and video frames that do not match the first text description;

computing a video-text contrastive loss by taking a weighted sum of the first and the second contrastive losses; and

updating the video encoder and the text encoder based at least in part on the video-text contrastive loss.

18. The processor-readable non-transitory storage medium of claim 17 , wherein the plurality of video frames are sparsely sampled from a video stream.

19. The processor-readable non-transitory storage medium of claim 17 , wherein the operations further comprise:

partitioning each video frame into a number of non-overlapping patches;

feeding the number of non-overlapping patches to a linear projection layer to produce a sequence of patch tokens;

appending a video start token to the sequence of patch tokens.

20. The processor-readable non-transitory storage medium of claim 17 , wherein an operation of encoding, by the video encoder, the plurality of video frames into video feature representations further comprises:

applying self-attention along a temporal dimension and a spatial dimension to an input sequence of tokens to result in per-frame features; and

applying a temporal fusion layer to the per-frame features along the temporal dimension to aggregate per-frame features into video features.