Systems and methods for contrastive pretraining with video tracking supervision

1. A method of pretraining a vision model, the method comprising:

receiving, via a communication interface, an input video;

extracting a first set of video frames from the input video subject to a temporal constraint, wherein each of the first set of video frames corresponds to a respective area of salient region that is non-zero;

generating a first set of saliency maps as tracking masks corresponding to the first set of video frames;

generating, a key crop and a query crop from the first set of video frames subject to a spatial constraint that the key crop and the query crop satisfy an intersection over union (IOU) threshold with a respective tracking mask from the tracking masks;

encoding, by a momentum encoder, the key crop into a key feature representation;

encoding, by an encoder of the vision model, the query crop into a query feature representation;

computing a contrastive loss based on the key feature representation and the query feature representation; and

updating the vision model based at least in part on the contrastive loss.

2. The method of claim 1 , wherein the first set of video frames are sampled by:

obtaining a saliency map for an initial frame of the input video; and

tracking a target object in the saliency map throughout the input video by checking an IOU constraint across continuous frame masks in the input video.

3. The method of claim 1 , wherein the key crop and the query crop sampled from the input video serve as a positive pair for contrastive learning, and

wherein the key crop paired with another query crop sampled from another different video form a negative pair for contrastive learning.

4. The method of claim 1 , further comprising:

sampling, from the input video, a first video segment and a second video segment having a same speed, and a third video segment having a different speed;

forming a positive input pair of the first video segment and the second video segment;

forming a negative input pair of the first video segment or the second video segment and the third video segment;

inputting the positive input pair and the negative input pair to the vision model;

computing a speed loss based on a difference between a distance between the positive input pair and the negative input pair in a feature space and a pre-defined margin; and

updating the vision model by a weighted sum of the contrastive loss and the speed loss.

5. The method of claim 1 , further comprising:

generating a key foreground of the key crop; and

generating, by the momentum encoder, a key foreground feature representation from the key foreground;

computing an attention heatmap based on the query feature representation and the key foreground feature representation.

6. The method of claim 5 , wherein the attention heatmap is computed based on a linear combination of a product between a last convolutional layer activation of the encoder and an importance metric of the last convolution layer of the encoder.

7. The method of claim 6 , wherein the importance metric of the last convolution layer of the encoder is computed based on the key foreground feature representation and gradients of the last convolutional layer activations of the encoder.

8. The method of claim 5 , further comprising:

generating a pseudo segmentation ground truth by averaging tracking masks corresponding to the query crop; and

computing an attention loss based on the attention heatmap and the pseudo segmentation ground truth.

9. The method of claim 8 , further comprising:

computing a weighted sum of the attention loss and the contrastive loss; and

updating the vision model based on the weighted sum when the vision model is an image model.

10. The method of claim 8 , further comprising:

computing a weighted sum of the attention loss, the contrastive loss and a speed loss computed based on samples sampled at different video speeds; and

updating the vision model based on the weighted sum when the vision model is a video model.

11. A system of pretraining a vision model, the system comprising:

a communication interface receiving an input video;

a memory storing a plurality of processor-executable instructions;

a processor executing the plurality of processor-executable instructions to perform operations comprising:

extracting a first set of video frames from the input video subject to a temporal constraint, wherein each of the first set of video frames corresponds to a respective area of salient region that is non-zero;

generating a first set of saliency maps as tracking masks corresponding to the first set of video frames;

generating, a key crop and a query crop from the first set of video frames subject to a spatial constraint that the key crop and the query crop satisfy an intersection over union (IOU) threshold with a respective tracking mask from the tracking masks;

encoding, by a momentum encoder, the key crop into a key feature representation;

encoding, by an encoder of the vision model, the query crop into a query feature representation;

computing a contrastive loss based on the key feature representation and the query feature representation; and

updating the vision model based at least in part on the contrastive loss.

12. The system of claim 11 , wherein the first set of video frames are sampled by:

obtaining a saliency map for an initial frame of the input video; and

tracking a target object in the saliency map throughout the input video by checking an IOU constraint across continuous frame masks in the input video.

13. The system of claim 11 , wherein the key crop and the query crop sampled from the input video serve as a positive pair for contrastive learning, and

wherein the key crop paired with another query crop sampled from another different video form a negative pair for contrastive learning.

14. The system of claim 11 , wherein the operations further comprise:

sampling, from the input video, a first video segment and a second video segment having a same speed, and a third video segment having a different speed;

forming a positive input pair of the first video segment and the second video segment;

forming a negative input pair of the first video segment or the second video segment and the third video segment;

inputting the positive input pair and the negative input pair to the vision model;

computing a speed loss based on a difference between a distance between the positive input pair and the negative input pair in a feature space and a pre-defined margin; and

updating the vision model by a weighted sum of the contrastive loss and the speed loss.

15. The system of claim 11 , wherein the operations further comprise:

generating a key foreground of the key crop; and

generating, by the momentum encoder, a key foreground feature representation from the key foreground;

computing an attention heatmap based on the query feature representation and the key foreground feature representation.

16. The system of claim 15 , wherein the attention heatmap is computed based on a linear combination of a product between a last convolutional layer activation of the encoder and an importance metric of the last convolution layer of the encoder, and

wherein the importance metric of the last convolution layer of the encoder is computed based on the key foreground feature representation and gradients of the last convolutional layer activations of the encoder.

17. The system of claim 15 , wherein the operations further comprise:

generating a pseudo segmentation ground truth by averaging tracking masks corresponding to the query crop; and

computing an attention loss based on the attention heatmap and the pseudo segmentation ground truth.

18. The system of claim 17 , wherein the operations further comprise:

computing a weighted sum of the attention loss and the contrastive loss; and

updating the vision model based on the weighted sum when the vision model is an image model.

19. The system of claim 17 , wherein the operations further comprise:

computing a weighted sum of the attention loss, the contrastive loss and a speed loss computed based on samples sampled at different video speeds; and

updating the vision model based on the weighted sum when the vision model is a video model.

20. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for pretraining a vision model, the instructions being executed by a processor to perform operations comprising:

receiving, via a communication interface, an input video;

extracting a first set of video frames from the input video subject to a temporal constraint, wherein each of the first set of video frames corresponds to a respective area of salient region that is non-zero;

generating a first set of saliency maps as tracking masks corresponding to the first set of video frames;

generating, a key crop and a query crop from the first set of video frames subject to a spatial constraint that the key crop and the query crop satisfy an intersection over union (IOU) threshold with a respective tracking mask from the tracking masks;

encoding, by a momentum encoder, the key crop into a key feature representation;

encoding, by an encoder of the vision model, the query crop into a query feature representation;

computing a contrastive loss based on the key feature representation and the query feature representation; and

updating the vision model based at least in part on the contrastive loss.