Systems and methods for video and language pre-training
Embodiments described a method of video-text pre-learning to effectively learn cross-modal representations from sparse video frames and text. Specifically, an align and prompt framework provides a video and language pre-training framework that encodes the frames and text independently using a transformer-based video encoder and a text encoder. A multi-modal encoder is then employed to capture cross-modal interaction between a plurality of video frames and a plurality of texts. The pre-training includes a prompting entity modeling that enables the model to capture fine-grained region-entity alignment.
1. A method of video-and-language alignment pretraining with entity prompts, the method comprising:
obtaining, via a data interface, a video frame, and a set of text entity prompts that identify a pre-defined set of candidate entities;
encoding, by a first video encoder, an augmented view of the video frame into a video feature representation;
encoding, by a first text encoder, the set of text entity prompts into a set of prompt feature representations corresponding to the pre-defined set of candidate entities, respectively;
computing, for each candidate entity in the pre-defined set of candidate entities, a respective similarity score between a respective prompt feature representation corresponding to the respective candidate entity and the video feature representation;
computing an entity pseudo label as a probability distribution among the pre-defined set of candidate entities based on respective similarity scores corresponding to the pre-defined set of candidate entities;
generating, by a video-language model, an entity prediction in response to an input of the video frame, comprising:
encoding, by a second video encoder of the video-language model, the video frame into a second video feature representation,
encoding, by a second text encoder of the video-language model, a text descriptions into a second text feature representation,
encoding, by a multi-modal video-text encoder of the video-language model, the second video feature representation and the second text feature representation into a set of multimodal embeddings, and
generating, by a classifier of the video-language model, the entity prediction from the set of multimodal embeddings;
computing a first loss based on a cross-entropy between the entity prediction and the entity pseudo label that is the probability distribution; and
updating the video-language model comprising the second video encoder, the second text encoder and the multi-modal video-text encode based at least in part on the first loss that is computed based at least in part on outputs of the first video encoder and the first text encoder.
2. The method of claim 1 , wherein each text entity prompt from the set of text entity prompts takes a pre-determined format identifying a frequent noun that is a candidate entity in a pretraining corpus.
3. The method of claim 1 , wherein the video feature representation comprises an embedding of a video start token.
4. The method of claim 1 , wherein the entity pseudo label is computed by taking a softmax-normalized similarity between the embedding of the video start token and the set of prompt feature representations.
5. The method of claim 1 , further comprising:
obtaining, via the data interface, a plurality of video frames and a plurality of text descriptions;
encoding, by a second video encoder, the plurality of video frames into video feature representations;
encoding, by a second text encoder, the plurality of text descriptions into text feature representations;
computing similarity scores between the video feature representations and the text feature representations; and
computing a second loss that is a contrastive loss based on the computed similarity scores.
6. The method of claim 5 , further comprising:
predicting, by the video-language model, masked input text tokens that are input to the second text encoder;
computing a third loss based on the predicted masked input text tokens.
7. The method of claim 6 , further comprising:
generating, by the video-language model, an alignment between a video input and a text input; and
computing a fourth loss based on the generated alignment.
8. The method of claim 7 , further comprising:
computing a weighted sum of the first loss, the second loss, the third loss and the fourth loss; and
updating the video-language model based on the weighted sum.
9. A system of video-and-language alignment pretraining with entity prompts, the system comprising:
a data interface obtaining a video frame and a set of text entity prompts that identify a pre-defined set of candidate entities;
a memory storing a plurality of processor-executable instructions; and
a processor executing the plurality of processor-executable instructions to perform operations comprising:
encoding, by a first video encoder, an augmented view of the video frame into a video feature representation;
encoding, by a first text encoder, the set of text entity prompts into a set of prompt feature representations corresponding to the pre-defined set of candidate entities;
computing, for each candidate entity in the pre-defined set of candidate entities, a respective similarity score between a respective prompt feature representation corresponding to the respective candidate entity and the video feature representation;
computing an entity pseudo label as a probability distribution among the pre-defined set of candidate entities based on respective similarity scores corresponding to the pre-defined set of candidate entities;
generating, by a video-language model, an entity prediction in response to an input of the video frame, comprising:
encoding, by a second video encoder of the video-language model, the video frame into a second video feature representation,
encoding, by a second text encoder of the video-language model, a text descriptions into a second text feature representation,
encoding, by a multi-modal video-text encoder of the video-language model, the second video feature representation and the second text feature representation into a set of multimodal embeddings, and
generating, by a classifier of the video-language model, the entity prediction from the set of multimodal embeddings;
computing a first loss based on a cross-entropy between the entity prediction and the entity pseudo label that is the probability distribution; and
updating the video-language model comprising the second video encoder, the second text encoder and the multi-modal video-text encode based at least in part on the first loss that is computed based at least in part on outputs of the first video encoder and the first text encoder.
10. The system of claim 9 , wherein each text entity prompt from the set of text entity prompts takes a pre-determined format identifying a frequent noun that is a candidate entity in a pretraining corpus.
11. The system of claim 9 , wherein the video feature representation comprises an embedding of a video start token.
12. The system of claim 9 , wherein the entity pseudo label is computed by taking a softmax-normalized similarity between the embedding of the video start token and the set of prompt feature representations.
13. The system of claim 9 , wherein the operations further comprise:
obtaining, via the data interface, a plurality of video frames and a plurality of text descriptions;
encoding, by a second video encoder, the plurality of video frames into video feature representations;
encoding, by a second text encoder, the plurality of text descriptions into text feature representations;
computing similarity scores between the video feature representations and the text feature representations; and
computing a second loss that is a contrastive loss based on the computed similarity scores.
14. The system of claim 13 , wherein the operations further comprise:
predicting, by the video-language model, masked input text tokens that are input to the second text encoder;
computing a third loss based on the predicted masked input text tokens.
15. The system of claim 14 , wherein the operations further comprise:
generating, by the video-language model, an alignment between a video input and a text input;
computing a fourth loss based on the generated alignment;
computing a weighted sum of the first loss, the second loss, the third loss and the fourth loss; and
updating the video-language model based on the weighted sum.
16. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for video-and-language alignment pretraining with entity prompts, the instructions being executed by a processor to perform operations comprising:
obtaining, via a data interface, a video frame and a set of text entity prompts that identify a pre-defined set of candidate entities;
encoding, by a first video encoder, an augmented view of the video frame into a video feature representation;
encoding, by a first text encoder, the set of text entity prompts into a set of prompt feature representations that is a candidate entity;
computing, for each candidate entity in the pre-defined set of candidate entities, a respective similarity score between a respective prompt feature representation corresponding to the respective candidate entity and the video feature representation;
computing an entity pseudo label as a probability distribution among the pre-defined set of candidate entities based on respective similarity scores corresponding to the pre-defined set of candidate entities;
generating, by a video-language model, an entity prediction in response to an input of the video frame, comprising:
encoding, by a second video encoder of the video-language model, the video frame into a second video feature representation,
encoding, by a second text encoder of the video-language model, a text descriptions into a second text feature representation,
encoding, by a multi-modal video-text encoder of the video-language model, the second video feature representation and the second text feature representation into a set of multimodal embeddings, and
generating, by a classifier of the video-language model, the entity prediction from the set of multimodal embeddings;
computing a first loss based on a cross-entropy between the entity prediction and the entity pseudo label that is the probability distribution; and
updating the video-language model comprising the second video encoder, the second text encoder and the multi-modal video-text encode based at least in part on the first loss that is computed based at least in part on outputs of the first video encoder and the first text encoder.