IP Library › Granted Patent US 12664618

Granted Patent B2

US 12664618 · App. 17/720,845 · Granted Jun 23, 2026

Techniques for denoising videos

Inventors: Yang Zhang (Dubendorf, CH); Tunc Ozan Aydin (Zurich, CH); Christopher Richard Schroers (Uster, CH)

Assignee: DISNEY ENTERPRISES, INC.

G06T5/70 G06N20/00 G06T5/73 G06T2207/20081

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12664618
App. No.: 17/720,845
Granted: Jun 23, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Techniques are disclosed for denoising videos. In some embodiments, video frames are denoised using a denoising model that includes an encoder-decoder architecture and attention modules. During training of the denoising model, the attention modules learn weightings to upweight certain dimensions of input features to help pixel registration, remove ghosting artifacts, and improve temporal consistency when the frames of a video are being denoised. The denoising model can also be used to train a student denoising model that has a same architecture as, but is smaller and faster than, the denoising model. After training, noisy video frames can be input into the denoising model and/or the student denoising model to generate corresponding denoised video frames.

Claims (53)

1 . A computer-implemented method for denoising videos, the method comprising:

receiving a video that includes a plurality of frames; and

processing one or more frames included in the plurality of frames using a denoising model that includes an encoder and a decoder to generate one or more denoised frames,

wherein the encoder includes one or more encoder blocks and the decoder includes one or more decoder blocks, and each of the one or more encoder blocks and the one or more decoder blocks includes a three-dimensional (3D) convolutional layer and an attention module that calculates (i) a first set of weights for a plurality of feature channels associated with a plurality of features and (ii) a second set of weights for a spatial dimension associated with the plurality of features.

2 . The computer-implemented method of claim 1 , wherein an output of the 3D convolutional layer included in each of the one or more encoder blocks and each of the one or more decoder blocks is multiplied with another output of a corresponding attention module to generate the first set of weights and the second set of weights.

3 . The computer-implemented method of claim 1 , wherein the first set of weights and the second set of weights calculated by the attention module in a first encoder block included in the one or more encoder blocks are added to a corresponding plurality of features to generate a plurality of weighted features that are input into a second encoder block included in the one or more encoder blocks.

4 . The computer-implemented method of claim 1 , further comprising:

receiving a control map specifying an amount of denoising to perform; and

processing the one or more frames further based on the control map.

5 . The computer-implemented method of claim 1 , further comprising:

receiving a control map specifying an amount of sharpening to perform; and

processing the one or more frames further based on the control map.

6 . The computer-implemented method of claim 1 , further comprising:

adding a plurality of amounts of noise to a set of frames to generate a first plurality of sets of noisy frames, wherein each set of noisy frames includes a different amount of noise; and

training the denoising model based on the first plurality of sets of noisy frames.

7 . The computer-implemented method of claim 6 , further comprising:

adding a plurality of amounts of blur to the set of frames to generate a plurality of sets of blurry frames, wherein each set of blurry frames includes a different amount of blur;

adding noise to the plurality of sets of blurry frames to generate a second plurality of sets of noisy frames; and

training the denoising model based on the second plurality of sets of noisy frames.

8 . The computer-implemented method of claim 1 , further comprising:

training a second denoising model that is larger than the denoising model; and

training the denoising model based on a feature loss computed between distilled information from an adaptation layer of the second denoising model and feature maps from the denoising model.

9 . The computer-implemented method of claim 1 , further comprising training the denoising model based on a loss function that accounts for (i) a reconstruction difference between one or more additional denoised frames output by the denoising model and one or more reference frames; and (ii) a temporal consistency between the one or more additional denoised frames.

10 . One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for denoising videos, the steps comprising:

receiving a video that includes a plurality of frames; and

processing one or more frames included in the plurality of frames using a denoising model that includes an encoder and a decoder to generate one or more denoised frames,

11 . The one or more non-transitory computer-readable storage media of claim 10 , wherein an output of the 3D convolutional layer included in each of the one or more encoder blocks and each of the one or more decoder blocks is multiplied with another output of a corresponding attention module to generate the first set of weights and the second set of weights.

12 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the encoder encodes the one or more frames to generate one or more latent representations and the decoder decodes the one or more latent representations to generate the one or more denoised frames.

13 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the instructions, when executed by one or more processing units, further cause the one or more processing units to perform the steps of:

receiving a control map specifying an amount of denoising to perform; and

processing the one or more frames further based on the control map.

14 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the instructions, when executed by one or more processing units, further cause the one or more processing units to perform the steps of:

receiving a control map specifying an amount of sharpening to perform; and

processing the one or more frames further based on the control map.

15 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the instructions, when executed by one or more processing units, further cause the one or more processing units to perform the steps of:

adding a plurality of amounts of noise to a set of frames to generate a plurality of sets of noisy frames, wherein each set of noisy frames includes a different amount of noise; and

training the denoising model based on the plurality of sets of noisy frames.

16 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the instructions, when executed by one or more processing units, further cause the one or more processing units to perform the steps of:

adding a plurality of amounts of blur to a set of frames to generate a plurality of sets of blurry frames, wherein each set of blurry frames includes a different amount of blur;

adding noise to the plurality of sets of blurry frames to generate a plurality of sets of noisy frames; and

training the denoising model based on the plurality of sets of noisy frames.

17 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the instructions, when executed by one or more processing units, further cause the one or more processing units to perform the steps of:

training a second denoising model; and

training the denoising model based on a feature loss computed between distilled information from an adaptation layer of the second denoising model and feature maps from the denoising model.

18 . The one or more non-transitory computer-readable storage media of claim 10 , wherein the instructions, when executed by one or more processing units, further cause the one or more processing units to perform the step of training the denoising model based on a loss function that accounts for (i) a reconstruction difference between one or more additional denoised frames output by the denoising model and one or more reference frames; and (ii) a temporal consistency between the one or more additional denoised frames.

19 . A system, comprising:

one or more memories storing instructions; and

one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:

receive a video that includes a plurality of frames, and

process one or more frames included in the plurality of frames using a denoising model that includes an encoder and a decoder to generate one or more denoised frames,

20 . The system of claim 19 , wherein an output of the 3D convolutional layer included in each of the one or more encoder blocks and each of the one or more decoder blocks is multiplied with another output of a corresponding attention module to generate the first set of weights and the second set of weights.