IP Library › Granted Patent US 12,400,341
Granted Patent B2
US 12,400,341 · App. 17/570,254 · Granted Aug 26, 2025

Machine learning framework applied in a semi-supervised setting to perform instance tracking in a sequence of image frames

Inventors: Yang Fu (Urbana, IL); Sifei Liu (Santa Clara, CA); Umar Iqbal (San Jose, CA); Shalini De Mello (San Francisco, CA); Jan Kautz (Lexington, MA)
Assignee: NVIDIA Corporation
G06T7/248G06T7/11G06T7/73G06V10/764G06V10/7715G06V10/774G06V10/82G06T2207/10016G06T2207/20016G06T2207/20081G06T2207/20084G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,400,341
App. No.
17/570,254
Granted
Aug 26, 2025
Kind
B2
Abstract

A method and system are provided for tracking instances within a sequence of video frames. The method includes the steps of processing an image frame by a backbone network to generate a set of feature maps, processing the set of feature maps by one or more prediction heads, and analyzing the embedding features corresponding to a set of instances in two or more image frames of the sequence of video frames to establish a one-to-one correlation between instances in different image frames. The one or more prediction heads includes an embedding head configured to generate a set of embedding features corresponding to one or more instances of an object identified in the image frame. The method may also include training the one or more prediction heads using a set of annotated image frames and/or a plurality of sequences of unlabeled video frames.

Claims (60)

1. A method for tracking instances of objects within a sequence of video frames, the method comprising:

processing a first frame from the sequence of video frames by a backbone network to generate a first set of feature maps;

processing a second frame from the sequence of video frames by the backbone network to generate a second set of feature maps;

generating, by an embedding head, a first set of embedding vectors, wherein the embedding head generates the first set of embedding vectors by processing the first set of feature maps, wherein each embedding vector of the first set of embedding vectors corresponds to an instance of an object identified in the first frame;

generating, by the embedding head, a second set of embedding vectors, wherein the embedding head generates the second set of embedding vectors by processing the second set of feature maps, wherein each embedding vector of the second set of embedding vectors corresponds to an instance of an object identified in the second frame; and

generating, by a prediction head, a first predicted heatmap of keypoint locations for the first frame and a second predicted heatmap of keypoint locations for the second frame, wherein the prediction head generates the first predicted heatmap by processing the first set of feature maps and the prediction head generates the second predicted heatmap by processing the second set of feature maps,

wherein the embedding head comprises a keypoint embedding head including an encoder-decoder structure, wherein the encoder-decoder structure includes an encoder comprising a convolutional layer and a decoder comprising a de-convolutional layer.

2. The method of claim 1 , further comprising processing the set of feature maps by a classification head and a shape regression head configured to provide a pose estimation for each instance of an object based on a plurality of keypoints.

3. The method of claim 1 , further comprising:

concatenating the first predicted heatmap of keypoint locations and the first set of feature maps to produce first keypoint embedding head input;

concatenating the second predicted heatmap of keypoint locations and the second set of feature maps to produce second keypoint embedding head input;

generating, by the keypoint embedding head by processing the first keypoint embedding head input, a first set of keypoint embedding vectors;

generating, by the keypoint embedding head by processing the second keypoint embedding head input, a second set of keypoint embedding vectors; and

analyzing the first set of keypoint embedding vectors and the second set of keypoint embedding vectors to perform pose tracking of an instance of an object.

4. A method for tracking instances of objects within a sequence of video frames, the method comprising:

processing a first frame from the sequence of video frames by a backbone network to generate a first set of feature maps;

processing a second frame from the sequence of video frames by the backbone network to generate a second set of feature maps;

generating, by an embedding head, a first set of embedding vectors, wherein the embedding head generates the first set of embedding vectors by processing the first set of feature maps, wherein each embedding vector of the first set of embedding vectors corresponds to an instance of an object identified in the first frame;

generating, by the embedding head, a second set of embedding vectors, wherein the embedding head generates the second set of embedding vectors by processing the second set of feature maps, wherein each embedding vector of the second set of embedding vectors corresponds to an instance of an object identified in the second frame; and

comparing a first center representation, obtained by averaging embedding vectors from the first set of embedding vectors, to a second center representation, obtained by averaging embedding vectors from the second set of embedding vectors, to establish a one-to-one correlation between the instance of the object in the first frame and the instance of the object in the second frame.

5. The method of claim 4 , further comprising:

predicting, by a classification head, a location of instances of objects in the first frame and the second frame; and

predicting, by a mask head, a pixel level segmentation mask for each instance of an object identified in the first frame and for each instance of an object identified in the second frame.

6. The method of claim 4 , wherein the backbone network comprises a feature pyramid network, wherein the first set of feature maps comprises a first plurality of feature maps of different spatial resolutions, and wherein the second set of feature maps comprises a second plurality of feature maps of different spatial resolutions.

7. The method of claim 4 , further comprising training the embedding head using a set of annotated image frames and/or a plurality of sequences of unlabeled video frames.

8. The method of claim 7 , wherein training the embedding head comprises minimizing an instance contrastive loss term.

9. The method of claim 8 , wherein training the embedding head further comprises enforcing maximum entropy regularization for a similarity matrix.

10. The method of claim 7 , wherein training the embedding head comprises minimizing a cycle loss term calculated based on a forward affinity matrix and a reverse affinity matrix corresponding to a sequence of video frames.

11. The method of claim 4 , wherein the comparing the first center representation to the second center representation comprises computing a similarity value for the first center representation and the second center representation.

12. The method of claim 11 , wherein the similarity value is a cosine similarity value.

13. A system for tracking instances of objects within a sequence of video frames, comprising:

a non-transitory computer-readable memory; and

at least one processor configured to:

implement a plurality of neural networks including:

a backbone network configured to:

process a first frame from the sequence of video frames to generate a first set of feature maps, and

process a second frame from the sequence of video frames to generate a second set of feature maps, and

an embedding head configured to:

generate, by processing the first set of feature maps, a first set of embedding vectors, each embedding vector of the first set of embedding vectors corresponding to an instance of an object identified in the first frame, and

generate, by processing the second set of feature maps, a second set of embedding vectors, each embedding vector of the second set of embedding vectors corresponding to an instance of an object identified in the second frame, and

compare a first center representation, obtained by averaging embedding vectors from the first set of embedding vectors, to a second center representation, obtained by averaging embedding vectors from the second set of embedding vectors, to establish a one-to-one correlation between the instance of the object in the first frame and the instance of the object in the second frame.

14. The system of claim 13 , wherein the plurality of neural networks further comprises:

a classification head configured to predict a location of instances of objects in the first frame and the second frame; and

a mask head configured to predict a pixel level segmentation mask for each instance of an object identified in the first frame and for each instance of an object identified in the second frame.

15. The system of claim 13 , wherein the embedding head comprises a keypoint embedding head including an encoder-decoder structure, wherein the encoder-decoder structure includes an encoder comprising a convolutional layer and a decoder comprising a de-convolutional layer.

16. The system of claim 15 , wherein the plurality of neural networks further comprises a classification head and a shape regression head configured to estimate a pose estimation for each instance of an object based on a plurality of keypoints.

17. The system of claim 13 , wherein the backbone network comprises a feature pyramid network, wherein the first set of feature maps comprises a first plurality of feature maps of different spatial resolutions, and wherein the second plurality of feature maps comprises a second plurality of feature maps of different spatial resolutions.

18. The system of claim 13 , wherein the embedding head is trained using a set of annotated images and/or a plurality of sequences of unlabeled video frames.

19. The system of claim 13 , wherein training the embedding head comprises minimizing an instance contrastive loss term and enforcing maximum entropy regularization for a similarity matrix.

20. The system of claim 13 , wherein training the embedding head comprises minimizing a cycle loss term calculated based on a forward affinity matrix and a reverse affinity matrix corresponding to a sequence of video frames.

21. The system of claim 13 , wherein the at least one processor comprises a parallel processing unit.

22. The system of claim 13 , wherein the system is included in an autonomous vehicle.

23. The system of claim 13 , wherein the at least one processor is configured to compare the first center representation to the second center representation by computing a similarity value for the first center representation and the second center representation.

24. A non-transitory computer-readable medium storing instructions that, responsive to execution by one or more processors, cause a computing system to track instances of objects within a sequence of video frames by:

processing a first frame from the sequence of video frames by a backbone network to generate a first set of feature maps;

processing a second frame from the sequence of video frames by the backbone network to generate a second set of feature maps;

generating, by an embedding head, a first set of embedding vectors, wherein the embedding head generates the first set of embedding vectors by processing the first set of feature maps, wherein each embedding vector of the first set of embedding vectors corresponds to an instance of an object identified in the first frame;

generating, by the embedding head, a second set of embedding vectors, wherein the embedding head generates the second set of embedding vectors by processing the second set of feature maps, wherein each embedding vector of the second set of embedding vectors corresponds to an instance of an object identified in the second frame; and

comparing a first center representation, obtained by averaging embedding vectors from the first set of embedding vectors, to a second center representation, obtained by averaging embedding vectors from the second set of embedding vectors, to establish a one-to-one correlation between the instance of the object in the first frame and the instance of the object in the second frame.

25. The non-transitory computer-readable medium of claim 24 , wherein the comparing the first center representation to the second center representation comprises computing a similarity value for the first center representation and the second center representation.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2022
From: FU, YANG; LIU, SIFEI; IQBAL, UMAR; DE MELLO, SHALINI; KAUTZ, JAN
To: NVIDIA CORPORATION
Reel/Frame 058588/0468 →
Continuity (2)
Provisional Application 63135478 · Jan 8, 2021
Related Publication 20220222832A1 · Jul 14, 2022
References Cited (57)
US 11144726B2 · Chatterjee · 2021 [cited by examiner]
US 20160042227A1 · Zhong · 2016 [cited by examiner]
US 20170061215A1 · Son · 2017 [cited by examiner]
US 20190297326A1 · Reda et al. · 2019 [cited by applicant]
US 20200065976A1 · Han et al. · 2020 [cited by applicant]
US 20200202533A1 · Cohen · 2020 [cited by examiner]
US 20210012116A1 · Urtasun · 2021 [cited by examiner]
US 20210133987A1 · Suzuki · 2021 [cited by examiner]
US 20220222832A1 · Fu · 2022 [cited by examiner]
US 20220383630A1 · Beyer · 2022 [cited by examiner]
US 20230127865A1 · Shafik · 2023 [cited by examiner]
US 20230229734A1 · Troiani · 2023 [cited by examiner]
US 20230281979A1 · Jia · 2023 [cited by examiner]
US 20230337636A1 · Shmigelsky · 2023 [cited by examiner]
US 20230401827A1 · Kuen · 2023 [cited by examiner]
US 20240012852A1 · Mopur · 2024 [cited by examiner]
US 20240020963A1 · Qian · 2024 [cited by examiner]
US 20240070868A1 · Yu · 2024 [cited by examiner]
US 20240185617A1 · Brems · 2024 [cited by examiner]
KR 20200023221A · 2020 [cited by applicant]
Andriluka, M., “Posetrack: A benchmark for human pose estimation and tracking,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5167-5176, 2018. [cited by applicant]
Andriluka, M., et al., “2d human pose estimation: New benchmark and state of the art analysys,” In Proceedigns of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3686-3693, 2014. [cited by applicant]
Deng, J., et al., “Imagenet: A large-scale hierarchical image database,” In IEEE Conf. Comput. Vis. Pattern Recog., 2009. [cited by applicant]
Fang, H., et al., “RMPE: Regional multi-person estimation,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 2334-2343, 2017. [cited by applicant]
He, K., et al., “Deep residual learning for image recognition,” In IEEE Conf. Comput. Vis. Pattern Recog., pp. 770-778, 2016. [cited by applicant]
Lin, T., et al., “Feature pyramid networks for object detection,” In IEEE Conf. Comput. Vis. Pattern Recog., 2017. [cited by applicant]
Lin, T., et al., “Microsoft coco: Common objects in context,” In Eur. Conf. Comput. Vis., 2014. [cited by applicant]
Ning, G., et al., “Lighttrack: A generic framework for online top-down human pose tracking,” In Proceedings of the IEEE/CVF Conference on Cmoputer Vision and Pattern Recognition Workshops, pp. 1034-1035, 2020. [cited by applicant]
Pang, J., et al., “Quasi-dense instance similarly learning,” arXiv preprint arXiv:2006.06664, 2020. [cited by applicant]
Sun, K., et al., “Deep high-resolution representation learning for human pose estimation,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5693-5703, 2019. [cited by applicant]
Wang, X., et al., “Solo: Segmenting objects by locations,” Eur. Conf. Comput. Vis., 2020. [cited by applicant]
Wei, F., et al., “Point-set anchors for object detection, instance segmentation and pose estimation,” Eur. Conf. Comput. Vis., 2020. [cited by applicant]
Xu, N., et al., “Youtube-vos: A large-scale video object segmentation benchmark,” arXiv preprint arXiv:1809.03327, 2018. [cited by applicant]
Yang, L., et al., “Video instance segmentation,” In Int. Conf. Comput. Vis., 2019. [cited by applicant]
Bochinski, E., “High-speed tracking-by-detection without using image information,” In 2017 14 [cited by applicant]
Cao, J., et al., “Sipmask: Spatial information preservation for fast instance segmentation,” Proc. Eur. Conf. on Comput. Vis., 2020. [cited by applicant]
Chen, K., et al., “Mmdetection: Open mmlab detection toolbox and benchmark,” arXiv preprint arXiv:1906.07155, 2019. [cited by applicant]
Chen, T. et al., “A simple framework for contrastive learning of visual representations,” arXiv preprint arXiv:2002.05709, 2020. [cited by applicant]
Dong, M., et al., “Temporal feature augmented network for video instance segmentation,” In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019. [cited by applicant]
Fernando, B., et al., “Self-supervised video representation learning with odd-one-out networks,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogntiion, pp. 3636-3645, 2017. [cited by applicant]
Girdhar, R., et al., “Detect-and-track: Efficient pose estimation in videos,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 350-359, 2018. [cited by applicant]
Grill, J., et al., “Bootstrap your own latent—a new approach to self-supervised learning,” Advances in Neural Information Processing Systems, 33, 2020. [cited by applicant]
Guo, H., “Multi-domain pose network for multi-person pose estimation and tracking,” In Proceedings of the European Conference on Computer Vision, 2018. [cited by applicant]
Hwang, J., et al., “Pose estimator and tracker using temporal flow maps for limbs,” In 2019 International Joint Conference on Neural Networks, pp. 1-8, 2019. [cited by applicant]
Jabri, A., et al., “Space-time correspondence as a constrastive random walk,” Advances in Neural Information Processing Systems, 33, 2020. [cited by applicant]
Khosla, P., et al., “Supervised contrastive learning,” arXiv preprint arXiv:2004.11362, 2020. [cited by applicant]
Kong, T., et al., “Fovebox: Beyond anchor-based object detector,” IEEE Transactions on Image Processing, pp. 7389-7398, 2020. [cited by applicant]
Li, X., et al., “Joint-task self-supervised learning for temporal correspondence,” In Advances in Neural Information Processing Systems, pp. 318-328, 2019. [cited by applicant]
Luiten, J., et al., “Video instance segmentation 2019: A winning approach for combined detection, segmentation, classification and tracking,” In Proceedings of the IEEE Int'l Conf. on Comput. Vis. Workshops, 2019. [cited by applicant]
Misra, I., et al., “Shuffle and learn: unsupervised learning using temporal order verification,” In Eur. Conf. on Comput. Vis., pp. 527-554, Springer, 2016. [cited by applicant]
Ning, G., et al., “A top-down approach to articulated human pose estimation and tracking,” In Proceedings of the Eur. Conf. on Comput. Vis., 2018. [cited by applicant]
Van Den Oord, A., et al., “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018. [cited by applicant]
Pont-Tuset, J., et al., “The 2017 davis challenge on video object segmentation,” arXiv:1704.00675, 2017. [cited by applicant]
Raaj, Y., et al., “Efficient online multi-person 2d pose tracking with recurrent spatio-temporal affinity fields,” In Proceedings of the IEEE Conf. on Comput. Vis. and Pattern Recognition, pp. 4620-4628, 2019. [cited by applicant]
Ruggero, M., et al., “Benchmarking and error diagnosis in multi-instance pose estimation,” In Proceedings of the IEEE Int'l Conf. on Comp. Vis., pp. 369-378, 2017. [cited by applicant]
Snower, M., et al., “15 keypoints is all you need,” In Proceedings of the IEEE/CVF Conf. on Comput. Vis. and Pattern Recognition, pp. 6738-6748, 2020. [cited by applicant]
Vyas, A., et al., “Out-of distribution detection using an ensemble of self- supervised leave-out classifiers,” In Proceedings of the Eur. Conf. on Comput. Vis., pp. 550-564, 2018. [cited by applicant]