Estimating flow vectors for occluded content in video sequences
Systems and methods estimate optical flow vectors for occluded pixels between frames of a video sequence. Regions of occluded pixels may be identified and a cause of their occlusion may be determined. Different estimation techniques may be applied based, at least in part, on the cause of occlusion to provide a lightweight, less resource intensive estimation of optical flow data. Optical flow vectors for pixels that are occluded due to movement out of a frame may be estimated using a first technique while optical flow vectors for pixels that are occluded due to foreground movement may be estimated using a second technique.
1 . A method, comprising:
determining a set of occluded pixels, at least one individual occluded pixel of the set of occluded pixels having a respective valid flow vector in a first frame of a video sequence and a respective invalid flow vector in a second frame of the video sequence;
determining that an object depicted using at least a first pixel of the set of occluded pixels has moved out of a frame boundary between the first frame and the second frame based at least on motion data corresponding to at least the first pixel indicating a direction that is outside of the frame boundary;
estimating, for at least the first pixel, a first estimated motion vector corresponding to a reversal of a backward-directed flow vector corresponding to at least the first pixel;
determining at least a second pixel of the set of occluded pixels is occluded due to foreground object motion; and
estimating, for at least the second pixel, a second estimated motion vector corresponding to motion data of one or more boundary pixels.
2 . The method of claim 1 , wherein the first estimated motion vector and the second estimated motion vector are used, at least in part, to generate an intermediate frame between the first frame and the second frame.
3 . The method of claim 1 , further comprising tracking at least one object using, at least in part, the first estimated motion vector and the second estimated motion vector.
4 . The method of claim 1 , further comprising:
determining, for at least the second pixel, the one or more boundary pixels,
wherein the estimating the second estimated motion vector includes executing a fast scanline technique using the motion data of the one or more boundary pixels.
5 . The method of claim 4 , further comprising:
determining the at least the second pixel is filled with a first motion vector;
comparing a cost between the first motion vector and a second motion vector;
determining that the second motion vector has a lower cost; and
replacing the first motion vector with the second motion vector,
wherein the second estimated motion vector is the second motion vector.
6 . The method of claim 5 , wherein the cost is based at least in part on at least one of a traveled distance, a warp cost, or an intensity difference.
7 . A system, comprising:
one or more processing units to:
determine a set of occluded pixels, at least one individual occluded pixel of the set of occluded pixels having a respective valid flow vector in a first frame of a video sequence and a respective invalid flow vector in a second frame of the video sequence;
determine that an object depicted using at least a first pixel of the set of occluded pixels has moved out of a frame boundary between the first frame and the second frame based at least on motion data corresponding to the first pixel indicating a direction that is outside of the frame boundary;
estimate, for at least the first pixel, a first estimated motion vector corresponding to a reversal of a backward-directed flow vector corresponding to at least the first pixel;
determine at least a second pixel of the set of occluded pixels is occluded due to foreground object motion; and
estimate, for at least the second pixel, a second estimated motion vector corresponding to motion data of one or more boundary pixels.
8 . The system of claim 7 , wherein the first estimated motion vector and the second estimated motion vector are used, at least in part, to generate an intermediate frame between the first frame and the second frame.
9 . The system of claim 7 , wherein the one or more processing units are further to:
track at least one object using, at least in part, the first estimated motion vector and the second estimated motion vector.
10 . The system of claim 7 , wherein the one or more processing units are further to:
determine, for at least the second pixel, the one or more boundary pixels,
wherein the estimating the second estimated motion vector includes executing a fast scanline technique using the motion data of the one or more boundary pixels.
11 . The system of claim 10 , wherein the one or more processing units are further to:
determine the at least the second pixel is filled with a first motion vector;
compare a cost between the first motion vector and a second motion vector;
determine that the second motion vector has a lower cost; and
replace the first motion vector with the second motion vector,
wherein the second estimated motion vector is the second motion vector.
12 . The system of claim 11 , wherein the cost is based at least in part on at least one of a traveled distance, a warp cost, or an intensity difference.
13 . The system of claim 7 , wherein the system is comprised in at least one of:
a human-machine interface system of an autonomous or semi-autonomous machine;
a system for performing conversational AI operations;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
14 . A processor comprising:
one or more processing units to estimate a first estimated motion vector for at least a first pixel of a set of occluded pixels, the first estimated motion vector corresponding to a reversal of a backward-directed flow vector corresponding to at least the first pixel, and a second estimated motion for vector for at least a second pixel of the set of occluded pixels, the second estimated motion vector corresponding to motion data of one or more boundary pixels, based at least on determining at least one individual occluded pixel of the set of occluded pixels having a respective valid flow vector in a first frame of a video sequence and a respective invalid flow vector in a second frame of the video sequence, determining that an object depicted using at least a first pixel of the set of occluded pixels has moved out of a frame boundary between the first frame and the second frame based at least on the first pixel indicates a direction that is outside of the frame boundary, and determining the second pixel of the set of occluded pixels is occluded due to foreground object motion.
15 . The processor of claim 14 , wherein the first estimated motion vector and the second estimated motion vector are used, at least in part, to generate an intermediate frame between the first frame and the second frame.
16 . The processor of claim 14 , wherein the one or more processing units are further to:
track at least one object using, at least in part, the first estimated motion vector and the second estimated motion vector.
17 . The processor of claim 14 , wherein the one or more processing units are further to:
determine, for at least the second pixel, the one or more boundary pixels,
wherein the estimating the second estimated motion vector includes executing a fast scanline technique using the motion data of the one or more boundary pixels.