IP Library › Granted Patent US 10,964,000
Granted Patent B2
US 10,964,000 · App. 16/279,668 · Granted Mar 30, 2021

Techniques for reducing noise in video

Inventors: Seungseok Oh (Fremont, CA); Sean Pieper (San Jose, CA)
Assignee: NVIDIA Corporation
G06T5/20G06T1/20H04N5/357
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,964,000
App. No.
16/279,668
Granted
Mar 30, 2021
Kind
B2
Abstract

Systems and techniques for noise reduction in video are described. Example implementations provide improved motion-adaptive temporal or spatio-temporal noise reduction that use an improved blending of the current frame with previous frames. The improved blending may be particularly effective for processing video captured in noisy environments such as low-light and/or mobile environments. In some example implementations, the improved blending is based on more accurately distinguishing between pixel difference in adjacent images that are caused by motion rather than noise.

Claims (43)

1. A method of video noise reduction comprising:

determining a difference metric for a frame of a sequence of video frames;

determining, using the determined difference metric, a blending factor for blending the frame with another frame of the sequence, wherein the determining the blending factor includes at least one of (a) determining the blending factor jointly for a plurality of color channels, (b) non-linearly associating the determined difference metric with the blending factor, (c) restricting the determined blending factor to a maximum value determined according to a noise variance in the another frame, (d) restricting the determined blending factor to a maximum value based on a previous blending factor, or (e) changing a value of the blending factor according to one or more dynamically configurable coefficients;

blending the frame and the another frame according to the determined blending factor;

spatially processing the frame to reduce noise; and

combining the spatially processed frame with the blended frame to generate an output frame for transmitting.

2. The method according to claim 1 , wherein the spatially processing comprises spatially processing using a bilateral filter.

3. The method according to claim 1 , wherein the combining the spatially processed frame with the blended frame is based upon the blending factor.

4. The method according to claim 1 , wherein the determining the blending factor includes determining the blending factor jointly for a plurality of color channels.

5. The method according to claim 4 , wherein the determining the blending factor further includes determining the blending factor jointly for each of red, green and blue color channels in an RGB color format of the sequence of video frames or for each of luma and chroma color channels in an YUV color format of the sequence of video frames.

6. The method according to claim 1 , wherein the determining the blending factor includes non-linearly associating the determined difference metric with the blending factor.

7. The method according to claim 6 , wherein the determining the blending factor further includes incorporating a quadratic or higher-order term of the determined difference factor in the determined blending factor.

8. The method according to claim 1 , wherein the determining the blending factor includes restricting the determined blending factor to a maximum value determined according to a noise variance in the another frame.

9. The method according to claim 8 , wherein the determining the blending factor includes restricting the determined blending factor to a maximum value determined according to a ratio of noise variances in the another frame and the frame.

10. The method according to claim 9 , further comprising accessing a memory to store and/or to retrieve the noise valiances in downsampled form and/or in lower precision form.

11. The method according to claim 1 , wherein the determining the blending factor includes restricting the determined blending factor to a maximum value based on a previous blending factor.

12. The method according to claim 11 , further comprising accessing a memory to store and/or to retrieve a previous blending factor in downsampled form and/or in lower precision form.

13. The method according to claim 12 , wherein the determining the blending factor includes restricting the determined blending factor to a maximum value based on said previous blending factor and a difference between the current blending factor and the previous blending factor.

14. The method according to claim 1 , wherein the determining the blending factor includes changing a value of the blending factor according to one or more dynamically configurable coefficients.

15. The method according to claim 14 , wherein the changing is performed automatically in response to detection of predetermined conditions in statistics determined by an image signal processor.

16. The method according to claim 1 , wherein determining the blending factor further includes clustering the input video frame according to statistics from an image signal processor and further determining the blending factor in accordance with the clustering.

17. The method according to claim 1 , wherein the determining the difference metric and the determining the blending factor are performed for only some pixels of the input video frame, and wherein the some pixels are determined according to a predetermined pattern configuration.

18. A system comprising at least one graphics processing unit (GPU) configured to receive a sequence of video frames from a video producer and perform video noise reduction operations comprising:

determining a difference metric for a frame of the sequence of video frames;

determining, using the determined difference metric, a blending factor for blending the frame with another frame of the sequence, wherein the determining the blending factor includes at least one of (a) determining the blending factor jointly for a plurality of color channels, (b) non-linearly associating the determined difference metric with the blending factor, (c) restricting the determined blending factor to a maximum value determined according to a noise variance in the another frame, (d) restricting the determined blending factor to a maximum value based on a previous blending factor, or (e) changing a value of the blending factor according to one or more dynamically configurable coefficients;

blending the frame and the another frame according to the determined blending factor;

spatially processing the frame to reduce noise; and

combining the spatially processed frame with the blended frame to generate an output frame for transmitting.

19. The system according to claim 18 , wherein the system is a system-on-chip further comprises an image signal processor.

20. A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor, causes the at least one processor to perform operations comprising:

determining a difference metric for a frame of a sequence of video frames;

determining, using the determined difference metric, a blending factor for blending the frame with another frame of the sequence, wherein the determining the blending factor includes (a) determining the blending factor jointly for a plurality of color channels and/or (b) non-linearly associating the determined difference metric with the blending factor and/or (c) restricting the determined blending factor to a maximum value determined according to a noise variance in the another frame and/or (d) restricting the determined blending factor to a maximum value based on a previous blending factor and/or (e) changing a value of the blending factor according to one or more dynamically configurable coefficients;

blending the frame and the another frame according to the determined blending factor;

spatially processing the frame to reduce noise; and

combining the spatially processed frame with the blended frame to generate an output frame for transmitting.

21. A method of video noise reduction comprising:

determining a difference metric for a frame of a sequence of video frames;

determining, using the determined difference metric, a blending factor for blending the frame with another frame of the sequence, wherein the determining the blending factor includes at least one of (a) non-linearly associating the determined difference metric with the blending factor and incorporating a quadratic or higher-order term of the determined difference metric in the determined blending factor, (b) restricting the determined blending factor to a maximum value determined according to a ratio of noise variance in the another frame and the frame, (c) restricting the determined blending factor to a maximum value based on a previous blending factor and accessing a memory to store and/or to retrieve a previous blending factor in downsampled form and/or in lower precision form, or (d) changing a value of the blending factor according to one or more dynamically configurable coefficients automatically in response to detection of predetermined conditions in statistics determined by an image signal processor; and

blending the frame and the another frame according to the determined blending factor.

22. A system comprising at least one graphics processing unit (GPU) configured to receive a sequence of video frames from a video producer and perform video noise reduction operations comprising:

determining a difference metric for a frame of a sequence of video frames;

determining, using the determined difference metric, a blending factor for blending the frame with another frame of the sequence, wherein the determining the blending factor includes at least one of (a) non-linearly associating the determined difference metric with the blending factor and incorporating a quadratic or higher-order term of the determined difference metric in the determined blending factor, (b) restricting the determined blending factor to a maximum value determined according to a ratio of noise variance in the another frame and the frame, (c) restricting the determined blending factor to a maximum value based on a previous blending factor and accessing a memory to store and/or to retrieve a previous blending factor in downsampled form and/or in lower precision form, or (d) changing a value of the blending factor according to one or more dynamically configurable coefficients automatically in response to detection of predetermined conditions in statistics determined by an image signal processor; and

blending the frame and the another frame according to the determined blending factor.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 21, 2019
From: OH, SEUNGSEOK; PIEPER, SEAN
To: NVIDIA CORPORATION
Reel/Frame 048400/0851 →
Continuity (2)
Provisional Application 62699214 · Jul 17, 2018
Related Publication 20200027202A1 · Jan 23, 2020
Cited By (1)
US 12,293,490