High-quality denoising of an image sequence

1. A computer-implemented method, comprising:

receiving an image sequence comprising a plurality of frames;

determining a first patch in a first frame of the image sequence, the first patch comprising a subset of image data in the first frame;

determining a size of a local window in each of a neighboring set of the plurality of frames in the image sequence based on a local noise level or a local gradient, the determined size of the local window being larger for a larger noise level or more gradient content or being smaller for a smaller noise level or less gradient content;

determining locations of a plurality of corresponding patches in the neighboring set of the plurality of frames in the image sequence based on the determined size of the local window;

determining for each of the plurality of corresponding patches in the neighboring set of the plurality of frames, a corresponding subset of image data; and

generating a denoised first patch, comprising averaging image data from the first patch with the corresponding image data from the plurality of corresponding patches in the neighboring set of the plurality of frames.

2. The method as recited in claim 1 , further comprising:

applying optical flow to the image sequence to determine pixel-wise motion information, wherein the determining the locations of the plurality of corresponding patches in the neighboring set of the plurality of frames is based on the optical flow.

3. The method as recited in claim 2 , wherein the local noise is statistically independent noise that is removable by applying the pixel-wise information determined by the optical flow.

4. The method as recited in claim 3 , further comprising:

determining a denoising parameter based on the local noise level, wherein the determining the size of the local window in each of the neighboring set of the plurality of frames is proportional to the determined denoising parameter.

5. The method as recited in claim 1 , further comprising:

obtaining for each of the locations of the plurality of corresponding patches, an occlusion map, the occlusion map based on the optical flow; and

excluding from the plurality of corresponding patches, a patch comprising an occluded pixel.

6. The method as recited in claim 1 , further comprising:

generating a denoised image sequence by generating a plurality of denoised patches for each of the plurality of frames in the image sequence; and

substituting the denoised patches for original patches in each of the plurality of frames in the image sequence.

7. The method as recited in claim 6 , further comprising:

generating an upsampled denoised frame based on a denoised frame in the denoised image sequence;

generating a smoothed denoised image sequence based on the denoised image sequence;

determining a plurality of patches in the upsampled denoised frame, wherein each of the plurality of patches comprises a subset of image data in the upsampled denoised frame;

determining locations of a plurality of corresponding patches in a neighboring set of the plurality of frames in the smoothed denoised image sequence;

generating a plurality of high-frequency patches, wherein each high-frequency patch is based on image data at the locations of the plurality of corresponding patches in the image sequence;

weighting each of the plurality of high-frequency patches based on a similarity of each of the high-frequency patches to the plurality of corresponding patches;

averaging the plurality of weighted high-frequency patches; and

adding the averaged plurality of high-frequency patches to the upsampled denoised frame.

8. A system, comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to:

receive an image sequence comprising a plurality of frames;

determine a first patch in a first frame of the image sequence, the first patch comprising a subset of image data in the first frame;

determine a size of a local window in each of a neighboring set of the plurality of frames in the image sequence based on a local noise level, the determined size of the local window being larger for a larger noise level or being smaller for a smaller noise level;

determine locations of a plurality of corresponding patches in the neighboring set of the plurality of frames in the image sequence based on the determined size of the local window;

determine for each of the plurality of corresponding patches in the neighboring set of the plurality of frames, a corresponding subset of image data;

generate a denoised first patch, comprising averaging image data from the first patch with the corresponding image data from the plurality of corresponding patches in the neighboring set of the plurality of frames.

9. The system as recited in claim 8 , wherein the program instructions are executable by the at least one processor to:

apply optical flow to the image sequence to determine pixel-wise motion information;

determine the locations of the plurality of corresponding patches in the neighboring set of the plurality of frames based on the optical flow.

10. The system as recited in claim 8 , wherein the program instructions are executable by the at least one processor to:

determine a denoising parameter based on the local noise level, wherein determining the size of the local window in each of the neighboring set of the plurality of frames is proportional to the determined denoising parameter.

11. The system as recited in claim 10 , wherein the program instructions are executable by the at least one processor to:

determine a gradient of the plurality of corresponding patches, wherein the size of the local window in each of the neighboring set of the plurality of frames is proportional to the determined gradient.

12. The system as recited in claim 8 , wherein, in determining the locations of the plurality of corresponding patches, the program instructions are executable by the at least one processor to further comprise:

obtain for each of the locations of the plurality of corresponding patches, an occlusion map, the occlusion map based on the optical flow; and

exclude from the plurality of corresponding patches, a patch comprising an occluded pixel.

13. The system as recited in claim 8 , wherein the program instructions are executable by the at least one processor to:

generate a denoised image sequence, comprising:

generating a plurality of denoised patches for each of the plurality of frames in the image sequence; and

substituting the denoised patches for original first patches in each of the plurality of frames in the image sequence.

14. The system as recited in claim 13 , wherein the program instructions are executable by the at least one processor to:

generate an upsampled denoised frame based on a denoised frame in the denoised image sequence;

generate a smoothed denoised image sequence based on the denoised image sequence;

determine a plurality of patches in the upsampled denoised frame, wherein each of the plurality of patches comprises a subset of image data in the upsampled denoised frame;

determine locations of a plurality of corresponding patches in a neighboring set of the plurality of frames in the smoothed de noised image sequence;

generate a plurality of high-frequency patches, wherein each high-frequency patch is based on image data at the locations of the plurality of corresponding patches in the image sequence;

weight each of the plurality of high-frequency patches based on a similarity of each of the high-frequency patches to the plurality of corresponding patches;

average the plurality of weighted high-frequency patches; and

add the averaged plurality of high-frequency patches to the upsampled denoised frame.

15. A system comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to:

receive an image sequence comprising a plurality of frames;

determine a first patch in a first frame of the image sequence, the first patch comprising a subset of image data in the first frame;

determine a size of a local window in each of a neighboring set of the plurality of frames in the image sequence based on a local gradient, the determined size of the local window being larger for more gradient content or being smaller for less gradient content;

determine locations of a plurality of corresponding patches in the neighboring set of the plurality of frames in the image sequence based on the determined size of the local window;

determine for each of the plurality of corresponding patches in the neighboring set of the plurality of frames, a corresponding subset of image data; and

generate a denoised first patch, comprising averaging image data from the first patch with the corresponding image data from the plurality of corresponding patches in the neighboring set of the plurality of frames.

16. The system as recited in claim 15 , further comprising:

apply optical flow to the image sequence to determine pixel-wise motion information; and

determine the locations of the plurality of corresponding patches in the neighboring set of the plurality of frames based on the optical flow.

17. The system as recited in claim 16 , wherein further comprising:

determine, in the local window in each of a neighboring set of the plurality of frames in the image sequence, a local noise that is statistically independent noise that can be removed by applying the pixel-wise information determined by the optical flow.

18. The system as recited in claim 17 , further comprising:

determine a denoising parameter based on the local noise level, wherein the determining the size of the local window in each of the neighboring set of the plurality of frames is also proportional to the determined denoising parameter.

19. The system as recited in claim 15 , further comprising:

generate a denoised image sequence by generating a plurality of denoised patches for each of the plurality of frames in the image sequence; and

substitute the denoised patches for original patches in each of the plurality of frames in the image sequence.

20. The system as recited in claim 19 , further comprising:

generate an upsampled denoised frame based on a denoised frame in the denoised image sequence;

generate a smoothed denoised image sequence based on the denoised image sequence;

determine a plurality of patches in the upsampled denoised frame, wherein each of the plurality of patches comprises a subset of image data in the denoised frame;

determine locations of a plurality of corresponding patches in a neighboring set of the plurality of frames in the denoised image sequence;

generate a plurality of high-frequency patches, wherein each high-frequency patch is based on image data at the locations of the plurality of corresponding patches in the image sequence;

weight each of the plurality of high-frequency patches based on a similarity of each of the high-frequency patches to the plurality of corresponding patches;

average the plurality of weighted high-frequency patches; and

add the averaged plurality of high-frequency patches to the upsampled denoised frame.