Video retrieval techniques using video contrastive learning
A method, computer system, and a computer program product are provided for training a neural network for finding queried videos. Two pairs of video clips and associated text are obtained from a first dataset and a second dataset. The first dataset is used to train two video encoders by providing the video clips to the encoders as input and providing the outputs to a cosine similarity calculator. The second dataset is used to train a multi-mentor paradigm with two mentors. A first mentor and a second mentor are each provided the pair of textual data inputs. The first mentor provides a similarity value comparison, and the second mentor provides a word mover distance. Using the output from the multi-mentor paradigm and the encoders, a contrastive loss is calculated and used to provide contrastive learning of video features by differentiating similarity and dissimilarity of the video clips.
1. A method of training a neural network for finding and retrieving queried videos, comprising:
obtaining two video clips from a first dataset and providing the two video clips to two video encoders for training;
providing an output of each of the two video encoders to a cosine similarity calculator;
training a multi-mentor paradigm having at least two mentors by obtaining two textual inputs from a second dataset, wherein a first mentor is provided each textual input to provide a similarity value comparison and a second mentor is provided said two textual inputs to provide a word mover distance (WMD); and
using said output from said multi-mentor paradigm and said encoders, calculate a contrastive loss used to provide contrastive learning of video features for differentiating similarity and dissimilarity of video clips.
2. The method of claim 1 , wherein said first dataset comprises pairs of video and text clips that are overlapping on a timeline.
3. The method of claim 1 , wherein said first dataset is used to construct a positive video and text pair for contrastive learning.
4. The method of claim 1 , wherein said second dataset comprises a corpus having pairs of substantially similar texts.
5. The method of claim 4 , wherein said corpus can include an online or an offline document.
6. The method of claim 1 , wherein said first and second mentor each identify a similarity between said two inputted texts from a perspective of text semantics.
7. The method of claim 1 , wherein said multi-mentor paradigm has a feedforward network with multiple layers, and said feedforward network combines a plurality of outputs from all mentors in order to generate a label.
8. The method of claim 1 , wherein said encoders use spatio-temporal tokens within each video to provide a video feature vector used in said cosine similarity calculator.
9. The method of claim 1 , wherein said encoders are part of a Siamese neural network.
10. The method of claim 9 , wherein a same weight is given to video clips provided to said encoders to compute a comparable output vector.
11. The method of claim 10 , wherein said weights are adjusted during said training.
12. The method of claim 1 , wherein said neural network is used to find and retrieve a queried video from a corpus based on a search request.
13. A computer system for training a neural network to find and retrieve queried videos, comprising:
one or more non-transitory computer readable storage media storing program instruction to be executed by a processor;
program instructions, stored on at least one of the one or more non-transitory computer-readable storage media for execution by at least one of the one or more processors via at least one of the one or more memories to update, wherein the computer system is enabled to perform the steps:
obtaining two video clips from a first dataset and providing it to two video encoders for training;
providing an output of each encoder and providing it to a cosine similarity calculator;
training a multi-mentor paradigm having at least two mentors by obtaining two textual inputs from a second dataset; wherein a first mentor is provided each textual input to provide a similarity value comparison and said second mentor is provided said two textual inputs to provide a word mover distance (WMD);
using said output from said multi-mentor paradigm and said encoders, calculate a contrastive loss used to provide contrastive learning of video features for differentiating similarity and dissimilarity of video clips.
14. The computer system of claim 13 , wherein said first dataset comprises pairs of video and text clips that are overlapping on a timeline and said second dataset comprises a corpus having pairs of substantially similar texts.
15. The computer system of claim 13 , wherein said first and second mentor each identify similarity between said two inputted texts from a perspective of text semantics.
16. The computer system of claim 13 , wherein said multi-mentor paradigm have a feedforward network with multiple layers and said feedforward network combines a plurality of outputs from all mentors in order to generate a label.
17. A computer program product for training a neural network for finding and retrieving queried videos, comprising:
one or more non-transitory computer readable storage media storing program instruction to be executed by a processor;
program instructions, stored on at least one of the one or more non-transitory computer-readable storage media, to update the program instructions comprising:
obtaining two video clips from a first dataset and providing it to two video encoders for training;
providing an output of each encoder and providing it to a cosine similarity calculator;
training a multi-mentor paradigm having at least two mentors by obtaining two textual inputs from a second dataset; wherein a first mentor is provided each textual input to provide a similarity value comparison and said second mentor is provided said two textual inputs to provide a word mover distance (WMD);
using said output from said multi-mentor paradigm and said encoders, calculate a contrastive loss used to provide contrastive learning of video features for differentiating similarity and dissimilarity of video clips.
18. The computer program product of claim 17 , wherein said first dataset comprises pairs of video and text clips that are overlapping on a timeline and said second dataset comprises a corpus having pairs of substantially similar texts.
19. The computer program product of claim 17 , wherein said first and second mentor each identify similarity between said two inputted texts from a perspective of text semantics.
20. The computer program product of claim 17 , wherein said multi-mentor paradigm have a feedforward network with multiple layers, and said feedforward network combines a plurality of outputs from all mentors in order to generate a label.