IP Library › Granted Patent US 12705765
Granted Patent B2
US 12705765 · App. 17/246,149 · Granted Aug 11, 2026

Object tracking using optical flow

Inventors: Aurobinda Maharana (Maharashtra, IN); Vignesh Ungrapalli (Maharashtra, IN); Abhijit Patait (Maharashtra, IN)
Assignee: NVIDIA Corporation
G06T7/269G06T7/248G06T2207/10016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12705765
App. No.
17/246,149
Granted
Aug 11, 2026
Kind
B2
Abstract

Apparatuses, systems, and techniques are presented to track objects represented in images or video data. In at least one embodiment, motion of one or more objects within a plurality of digital images is determined based, at least in part, on flow information corresponding to the one or more objects.

Claims (69)

1 . One or more processors, comprising:

circuitry to use one or more neural networks to:

receive a plurality of images depicting one or more objects;

determine one or more points of one or more regions within one or more images of the plurality of images;

assign each pixel of one or more pixels of the one or more images to one or more spatial groups based, at least in part, on at least one point of the one or more points and one or more similarity thresholds, wherein the one or more similarity thresholds comprise one or more motion-vector similarity criteria for assigning each of the one or more pixels to the one or more spatial groups;

generate one or more representative motion vectors based, at least in part, on the one or more spatial groups, wherein the one or more representative motion vectors are an average of motion vectors of the one or more pixels of the one or more spatial groups; and

identify motion of the one or more objects within the plurality of images based, at least in part, on the representative motion vectors.

2 . The one or more processors of claim 1 , wherein the spatial groups comprises motion vectors of individual pixels between a pair of consecutive images of the plurality of images, and wherein to generate the one or more representative motion vectors based, at least in part, on a group of motion vectors further comprises determining the representative motion vectors from motion vectors of individual pixels.

3 . The one or more processors of claim 2 , wherein the representative motion vector of one or more regions of interest (ROIs) corresponding to the one or more objects is determined, and wherein the circuitry is further to apply the representative motion vectors to the one or more ROIs from a prior image of the pair to generate a predicted ROI of a current image of the pair, wherein the representative motion vectors are determined based on motion vectors of one or more subset of connected pixels within the one or more ROIs which have motion vectors satisfying a similarity threshold, wherein the one or more subset of connected pixels within the one or more ROIs corresponds to the one or more objects.

4 . The one or more processors of claim 3 , wherein the circuitry is further to perform a matching of one or more current ROIs from the current image with the one or more predicted ROIs, wherein labels of the one or more predicted ROIs are able to be associated with matching current ROIs from the current image.

5 . The one or more processors of claim 4 , wherein the matching involves a Hungarian matching process to minimize a cost for at least one of a centroid distance or an intersection over union between pairs of current and predicted ROIs.

6 . The one or more processors of claim 4 , wherein the circuitry is further to perform object identification to determine the one or more current ROIs for only a subset of images in the plurality of images.

7 . A system comprising:

one or more processors to use one or more neural networks to:

receive a plurality of images depicting one or more objects;

determine one or more points of one or more regions within one or more images of the plurality of images;

assign each pixel of one or more pixels of the one or more images to one or more spatial groups based, at least in part, on at least one point of the one or more points and one or more similarity thresholds, wherein the one or more similarity thresholds comprise one or more motion-vector similarity criteria for assigning each of the one or more pixels to the one or more spatial groups;

generate one or more representative motion vectors based, at least in part, on the one or more spatial groups, wherein the one or more representative motion vectors are an average of motion vectors of the one or more pixels of the one or more spatial groups; and

generate a representative motion vector based, at least in part, on a group of motion vectors corresponding to one or more objects in a subset of a plurality of images; and

identify motion of the one or more objects within the plurality of images based, at least in part, on the representative motion vectors.

8 . The system of claim 7 , wherein the group of motion vectors comprises motion vectors of individual pixels between a pair of consecutive images of the plurality of images, and wherein to generate the representative motion vectors based, at least in part, on a group of motion vectors further comprises determining the representative motion vectors from motion vectors of individual pixels.

9 . The system of claim 8 , wherein the representative motion vector of one or more regions of interest (ROIs) corresponding to the one or more objects is determined, and wherein the one or more processors are further to apply the representative motion vector to the one or more ROIs from a prior image of the pair to generate a predicted ROI of a current image of the pair, wherein the representative motion vectors are determined based on motion vectors of one or more subset of connected pixels within the one or more ROIs which have motion vectors satisfying a similarity threshold, wherein the one or more subset of connected pixels within the one or more ROIs corresponds to the one or more objects.

10 . The system of claim 9 , wherein the one or more processors are further to perform a matching of one or more current ROIs from the current image with the one or more predicted ROIs, wherein labels of the one or more predicted ROIs are able to be associated with matching current ROIs from the current image.

11 . The system of claim 10 , wherein the matching involves a Hungarian matching process to minimize a cost for at least one of a centroid distance or an intersection over union between pairs of current and predicted ROIs.

12 . The system of claim 10 , wherein the one or more processors are further to perform object identification to determine the one or more current ROIs for only a subset of images in the plurality of images.

13 . A method comprising:

using one or more neural networks to:

receive a plurality of images depicting one or more objects;

determine one or more points of one or more regions within one or more images of the plurality of images;

assign each pixel of one or more pixels of the one or more images to one or more spatial groups based, at least in part, on at least one point of the one or more points and one or more similarity thresholds, wherein the one or more similarity thresholds comprise one or more motion-vector similarity criteria for assigning each of the one or more pixels to the one or more spatial groups;

generate one or more representative motion vectors based, at least in part, on the one or more spatial groups; and

generate a representative motion vector based, at least in part, on a group of motion vectors corresponding to one or more objects in a subset of a plurality of digital images, wherein the one or more representative motion vectors are an average of the motion vectors of the one or more pixels of the one or more spatial groups; and

identify motion of the one or more objects within the plurality of digital images based, at least in part, on the representative motion vectors.

14 . The method of claim 13 , wherein the group of motion vectors comprises motion vectors of individual pixels between a pair of consecutive images of the plurality of digital images, and wherein to generate the representative motion vectors based, at least in part, on a group of motion vectors further comprises determining the representative motion vectors from motion vectors of individual pixels.

15 . The method of claim 14 , wherein the representative motion vector of one or more regions of interest (ROIs) corresponding to the one or more objects is determined, and further comprising:

applying the representative motion vector to the one or more ROIs from a prior image of the pair to generate a predicted ROI for a current image of the pair, wherein the representative motion vectors are determined based on motion vectors of one or more subset of connected pixels within the one or more ROIs which have motion vectors satisfying a similarity threshold, wherein the one or more subset of connected pixels within the one or more ROIs corresponds to the one or more objects.

16 . The method of claim 15 , further comprising:

performing a matching of one or more current ROIs from the current image with the one or more predicted ROIs, wherein labels of the one or more predicted ROIs are able to be associated with matching current ROIs from the current image.

17 . The method of claim 16 , wherein the matching involves a Hungarian matching process to minimize a cost for at least one of a centroid distance or an intersection over union between pairs of current and predicted ROIs.

18 . The method of claim 16 , further comprising:

performing object identification to determine the one or more current ROIs for only a subset of images in the plurality of digital images.

19 . A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:

use one or more neural networks to:

receive a plurality of images depicting one or more objects;

determine one or more points of one or more regions within one or more images of the plurality of images;

assign each pixel of one or more pixels of the one or more images to one or more spatial groups based, at least in part, on at least one point of the one or more points and one or more similarity thresholds, wherein the one or more similarity thresholds comprise one or more motion-vector similarity criteria for assigning each of the one or more pixels to the one or more spatial groups;

generate one or more representative motion vectors based, at least in part, on the one or more spatial groups, wherein the one or more representative motion vectors are an average of motion vectors of the one or more pixels of the one or more spatial groups; and

identify motion of the one or more objects within the plurality of images based, at least in part, on the representative motion vectors.

20 . The machine-readable medium of claim 19 , wherein the spatial groups comprises motion vectors of individual pixels between a pair of consecutive images of the plurality of images, and wherein to generate the representative motion vectors based, at least in part, on a group of motion vectors further comprises determining the representative motion vectors from motion vectors of individual pixels.

21 . The machine-readable medium of claim 20 , wherein the representative motion vector of one or more regions of interest (ROIs) corresponding to the one or more objects is determined, and wherein the instructions if performed further cause the one or more processors to:

apply the representative motion vector to the one or more ROIs from a prior image of the pair to generate a predicted ROI for a current image of the pair, wherein the representative motion vectors are determined based on motion vectors of one or more subset of connected pixels within the one or more ROIs which have motion vectors satisfying a similarity threshold, wherein the one or more subset of connected pixels within the one or more ROIs corresponds to the one or more objects.

22 . The machine-readable medium of claim 21 , wherein the instructions if performed further cause the one or more processors to:

perform a matching of one or more current ROIs from the current image with the one or more predicted ROIs, wherein labels of the one or more predicted ROIs are able to be associated with matching current ROIs from the current image.

23 . The machine-readable medium of claim 22 , wherein the matching involves a Hungarian matching process to minimize a cost of at least one of a centroid distance or an intersection over union between pairs of current and predicted ROIs.

24 . The machine-readable medium of claim 22 , wherein the instructions if performed further cause the one or more processors to:

perform object identification to determine the one or more current ROIs of only a subset of images in the plurality of images.

25 . An object tracking system, comprising:

one or more processors to use one or more neural networks to:

receive a plurality of images depicting one or more objects;

determine one or more points of one or more regions within one or more images of the plurality of images;

assign each pixel of one or more pixels of the one or more images to one or more spatial groups based, at least in part, on at least one point of the one or more points and one or more similarity thresholds, wherein the one or more similarity thresholds comprise one or more motion-vector similarity criteria for assigning each of the one or more pixels to the one or more spatial groups;

generate one or more representative motion vectors based, at least in part, on the one or more spatial groups, wherein the one or more representative motion vectors are an average of motion vectors of the one or more pixels of the one or more spatial groups; and

identify motion of the one or more objects within the plurality of images based, at least in part, on the representative motion vectors; and

memory to store data corresponding to the representative motion vectors.

26 . The object tracking system of claim 25 , wherein the spatial groups comprises motion vectors of individual pixels between a pair of consecutive images of the plurality of images, and wherein to generate the representative motion vectors based, at least in part, on a group of motion vectors further comprises determining the representative motion vectors from motion vectors of individual pixels.

27 . The object tracking system of claim 26 , wherein the representative motion vector of one or more regions of interest (ROIs) corresponding to the one or more objects is determined, and wherein the one or more processors are further to apply the representative motion vector to the one or more ROIs from a prior image of the pair to generate a predicted ROI for a current image of the pair, wherein the representative motion vector is determined based on motion vectors of one or more subset of connected pixels within the one or more ROIs which have motion vectors satisfying a similarity threshold, wherein the one or more subset of connected pixels within the one or more ROIs corresponds to the one or more objects.

28 . The object tracking system of claim 27 , wherein the one or more processors are further to perform a matching of one or more current ROIs from the current image with the one or more predicted ROIs, wherein labels of the one or more predicted ROIs are able to be associated with matching current ROIs from the current image.

29 . The object tracking system of claim 28 , wherein the matching involves a Hungarian matching process to minimize a cost for at least one of a centroid distance or an intersection over union between pairs of current and predicted ROIs.

30 . The object tracking system of claim 28 , wherein the one or more processors are further to perform object identification to determine the one or more current ROIs offer only a subset of images in the plurality of images.