Granted Patent B1

US 12,574,521 · App. 17/949,166 · Granted Mar 10, 2026

Inventors: Zhekun Luo (Santa Clara, CA); Robert Thomas Pottorff (Layton, UT); Karan Sapra (San Jose, CA); Jarmo Rafael Lunden (Espoo, FI); Andrew J. Tao (Los Altos, CA); Bryan Christopher Catanzaro (Los Altos Hills, CA)

Assignee: NVIDIA Corporation

H04N19/139G06T5/77G06T7/20G06T7/50H04N7/0127H04N19/132

Abstract

Claims (29)

1 . One or more processors, comprising: circuitry to:

identify one or more pixels not associated with any motion vectors and depicting at least a portion of one or more objects in one or more video frames;

identify one or more other pixels that are associated with one or more motion vectors, are within a predetermined proximity of the one or more pixels, have the same depth information as the one or more pixels, and also depict at least a portion of the one or more objects; and

use the one or more motion vectors associated with the one or more other pixels to estimate motion of the one or more pixels.

2 . The one or more processors of claim 1 , wherein the estimated motion of the one or more pixels comprises one or more forward motion vectors and the circuitry is to estimate the motion of the one or more pixels further based, at least in part, on one or more backward motion vectors.

3 . The one or more processors of claim 1 , wherein the circuitry is to estimate the motion of the one or more pixels by selecting the one or more motion vectors from one or more neighboring pixels based, at least in part, on the depth information of the one or more other pixels.

4 . The one or more processors of claim 1 , wherein the circuitry is to fill one or more holes of pixels using the estimated motion.

5 . The one or more processors of claim 1 , wherein the circuitry is to average the one or more motion vectors of one or more other pixels to estimate the motion.

6 . The one or more processors of claim 1 , wherein the circuitry is to estimate the motion further based, at least in part, on the depth information of the one or more other pixels.

7 . The one or more processors of claim 1 , wherein the circuitry is further to use the estimated motion to increase a framerate of a video.

8 . A computer-implemented method, comprising:

identifying one or more pixels not associated with any motion vectors and depicting at least a portion of one or more objects in one or more video frames;

identifying one or more other pixels that are associated with one or more motion vectors, are within a predetermined proximity of the one or more pixels, have the same depth information as the one or more pixels, and also depict at least a portion of the one or more objects; and

using the one or more motion vectors associated with the one or more other pixels to estimate motion of the one or more pixels.

9 . The computer-implemented method of claim 8 , wherein the one or more other pixels and the one or more pixels have depth information within a threshold range.

10 . The computer-implemented method of claim 8 , wherein the estimated motion of the one or more pixels comprises one or more motion vectors of a first direction and wherein the one or more motion vectors associated with the one or more other pixels used to estimate the motion of the one or more pixels comprise one or more motion vectors of a second direction different from the first direction.

11 . The computer-implemented method of claim 8 , further comprising downsampling one or more video frames and wherein the one or more pixels are in a video frame resulting from the downsampling.

12 . The computer-implemented method of claim 8 , wherein the motion of the one or more pixels indicates movement of the one or more pixels from a first video frame to a second video frame.

13 . The computer-implemented method of claim 8 , further comprising: using the estimated motion to generate pixel values of the one or more pixels.

14 . The computer-implemented method of claim 8 , wherein estimating the motion of the one or more pixels is part of generating the one or more video frames to be-between a first video frame and a second video frame of a video.

15 . A computer system, comprising: one or more processors and memory storing executable instructions that, if performed by the one or more processors, cause the computer system to:

identify one or more pixels not associated with any motion vectors and depicting at least a portion of one or more objects in one or more video frames;

identify one or more other pixels that are associated with one or more motion vectors, are within a predetermined proximity of the one or more pixels, the same depth information as the one or more pixels, and also depict at least a portion of the one or more objects; and

use the one or more motion vectors associated with the one or more other pixels to estimate motion of the one or more pixels.

16 . The computer system of claim 15 , wherein the one or more other pixels and the one or more other pixels have depth information within a threshold range.

17 . The computer system of claim 15 , wherein the estimated motion comprises one or more motion vectors of a first direction and is to be generated based further at least in part on one or more other motion vectors of a second direction different from the first direction.

18 . The computer system of claim 15 , wherein the executable instructions further cause the computer system to increase a framerate of a video based, at least in part, on the estimated motion.

19 . The computer system of claim 15 , wherein the executable instructions are to cause the motion to be estimated based further at least in part on one or more motion vectors provided from a game engine.

20 . The computer system of claim 15 , wherein the executable instructions are to further cause the computer system to perform an optical flow analysis to generate one or more motion vectors and wherein the executable instructions are to cause the computer system to estimate the motion based further at least in part on the one or more motion vectors.

Assignments (1)

References Cited (105)