Robust regression method for image-space denoising
The disclosure provides an approach for denoising (also referred to as “filtering”) rendered images. In one embodiment, a denoising application takes as input rendered images and feature buffers that encode image information such as surface positions, surface depths, surface normals, surface albedos, and distances to the camera. For each pixel in a received image, the denoising application performs a first-order regression in a predefined neighborhood of the pixel to find a linear combination of pixel features that fits pixel colors in the predefined neighborhood. In such a first-order regression, the local regression weight of each pixel in the neighborhood may be determined using a metric which computes distances based on color values in patches around pixels being compared. In another embodiment, collaborative filtering may be performed in which filtered output from the first-order regression in each neighborhood is averaged with filtered output from overlapping neighborhoods to obtain a final output.
1. A computer-implemented method for denoising a rendered image, comprising:
receiving the rendered image and a set of features associated with the rendered image;
for each pixel of a plurality of pixels in the rendered image, performing a regression in a predefined neighborhood of the pixel to find a combination of pixel features that fits pixel colors in the predefined neighborhood, wherein, in the regression, a respective local regression weight associated with each pixel in the predefined neighborhood is determined based, at least in part, on a distance metric which computes distances based, at least in part, on color values in patches around pixels being compared; and
determining a respective output color value for each pixel of a plurality of pixels in a denoised image corresponding to a respective one of the plurality of pixels in the rendered image based, at least in part, on the fittings produced by a plurality of the regressions.
2. The method of claim 1 , wherein the regression performed for each of the plurality of pixels in the rendered image is a first-order regression and the combination of pixel features that fits pixel colors in the predefined neighborhood is a linear combination.
3. The method of claim 1 , wherein:
the determining of the respective output color value for each pixel of the plurality of pixels in the denoised image includes averaging denoised color values of overlapping neighborhoods as produced by the plurality of regressions; and
in the averaging, each of the overlapping neighborhoods is weighted by a respective regression kernel determined using the distance metric which computes distances based, at least in part, on color values in patches around pixels being compared.
4. The method of claim 1 , further comprising, filtering the features to remove noise therein prior to performing the regression for each of the plurality of pixels in the rendered image.
5. The method of claim 3 , wherein:
colors of the received rendered image and the received features are separated into two statistically independent sets of buffers, each of the sets of buffers including a color buffer and a feature buffer, that are processed separately;
either features in each one of the sets of buffers are used in the regressions to fit colors of another one of the sets of buffers, or features in each one of the sets of buffers are used in the regressions to fit colors of the same set of buffers, or features and colors averaged on the two sets of buffers are used in the regressions to fit colors of both buffers, to determine respective intermediate output color values for each of the plurality of pixels in the rendered image; and
the intermediate output color values are combined to determine the respective output color value of each of the plurality of pixels in the rendered image.
6. The method of claim 5 , wherein a bandwidth controlling a number of pixels considered in the regression is determined based, at least in part, on squared differences between filtered color buffers of each one of the sets of buffers and the received color buffers of the other one of the sets of buffers.
7. The method of claim 5 , wherein:
the regression for each of the plurality of pixels in the rendered image is performed multiple times with different bandwidth parameters; and
the determined output color values are obtained as a weighted average of color buffers obtained from the multiple regressions with different bandwidth parameters.
8. The method of claim 1 , further comprising, repeating the step of performing the regression for each of a plurality of pixels in the denoised image having the output color values and the step of determining the respective output color values.
9. The method of claim 1 , wherein the set of features includes at least one of surface positions, surface depths, surface normals, surface albedos, distances to a camera, visibility, a constant feature, and horizontal and vertical ramp features.
10. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for denoising a rendered image, the operations comprising:
receiving the rendered image and a set of features associated with the rendered image;
for each pixel of a plurality of pixels in the rendered image, performing a regression in a predefined neighborhood of the pixel to find a combination of pixel features that fits pixel colors in the predefined neighborhood, wherein, in the regression, a respective local regression weight associated with each pixel in the predefined neighborhood is determined based, at least in part, on a distance metric which computes distances based, at least in part, on color values in patches around pixels being compared; and
determining a respective output color value for each pixel of a plurality of pixels in a denoised image corresponding to a respective one of the plurality of pixels in the rendered image based, at least in part, on the fittings produced by a plurality of the regressions.
11. The computer-readable storage medium of claim 10 , wherein the regression performed for each of the plurality of pixels in the rendered image is a first-order regression and the combination of pixel features that fits pixel colors in the predefined neighborhood is a linear combination.
12. The computer-readable storage medium of claim 10 , wherein:
the determining of the respective output color value for each pixel of the plurality of pixels in the denoised image includes averaging denoised color values of overlapping neighborhoods as produced by the plurality of regressions; and
in the averaging, each of the overlapping neighborhoods is weighted by a respective regression kernel determined using the distance metric which computes distances based, at least in part, on color values in patches around pixels being compared.
13. The computer-readable storage medium of claim 10 , the operations further comprising, filtering the features to remove noise therein prior to performing the regression for each of the plurality of pixels in the rendered image.
14. The computer-readable storage medium of claim 13 , wherein:
colors of the received rendered image and the received features are separated into two statistically independent sets of buffers, each of the sets of buffers including a color buffer and a feature buffer, that are processed separately;
either features in each one of the sets of buffers are used in the regressions to fit colors of another one of the sets of buffers, or features in each one of the sets of buffers are used in the regressions to fit colors of the same set of buffers, or features and colors averaged on the two sets of buffers are used in the regressions to fit colors of both buffers, to determine respective intermediate output color values for each of the plurality of pixels in the rendered image; and
the intermediate output color values are combined to determine the respective output color value of each of the plurality of pixels in the rendered image.
15. The computer-readable storage medium of claim 13 , wherein a bandwidth controlling a number of pixels considered in the regression is determined based, at least in part, on squared differences between filtered color buffers of each one of the sets of buffers and the received color buffers of the other one of the sets of buffers.
16. The computer-readable storage medium of claim 13 , wherein:
the regression for each of the plurality of pixels in the rendered image is performed multiple times with different bandwidth parameters; and
the determined output color values are obtained as a weighted average of color buffers obtained from the multiple regressions with different bandwidth parameters.
17. The computer-readable storage medium of claim 10 , the operations further comprising, repeating the step of performing the regression for each of a plurality of pixels in the denoised image having the output color values and the step of determining the respective output color values.
18. The computer-readable storage medium of claim 10 , wherein the set of features includes at least one of surface positions, surface depths, surface normals, surface albedos, distances to a camera, visibility, a constant feature, and horizontal and vertical ramp features.
19. A system, comprising:
a processor; and
a memory, wherein the memory includes an application program configured to perform operations for denoising a rendered image, the operations comprising:
receiving the rendered image and a set of features associated with the rendered image,
for each pixel of a plurality of pixels in the rendered image, performing a regression in a predefined neighborhood of the pixel to find a combination of pixel features that fits pixel colors in the predefined neighborhood, wherein, in the regression, a respective local regression weight associated with each pixel in the predefined neighborhood is determined based, at least in part, on a distance metric which computes distances based, at least in part, on color values in patches around pixels being compared, and
determining a respective output color value for each pixel of a plurality of pixels in a denoised image corresponding to a respective one of the plurality of pixels in the rendered image based, at least in part, on the fittings produced by a plurality of the regressions.
20. The system of claim 19 , wherein:
the regression performed for each of the plurality of pixels in the rendered image is a first-order regression and the combination of pixel features that fits pixel colors in the predefined neighborhood is a linear combination;
the determining of the respective output color value for each pixel of the plurality of pixels in the denoised image includes averaging denoised color values of overlapping neighborhoods as produced by the plurality of regressions; and
in the averaging, each of the overlapping neighborhoods is weighted by a respective regression kernel determined using the distance metric which computes distances based, at least in part, on color values in patches around pixels being compared.