Low-latency architecture for full frequency noise reduction in image processing
Systems and techniques provide for low-latency, full-frequency noise filtering of images through the use of an image-scaling-based filtering technique, or “multiscale filtering technique”, that can provide filtering for low, medium, and/or high frequencies for one or more components of an image, such that the different resolution scales at each level of the multiscale filtering technique provides a larger receptive field for a denoising process employed at each level than a conventional denoising framework. This multiscale filtering includes receiving an input image to be filtered and then performing a multiscale filtering process in which an input image is, at different resolution scales, denoised, downscaled, upscaled, and fused with a result of a lower resolution scale, to generate a filtered image. This may include temporarily buffering intermediate image data for some of the resolution scales at a memory using direct memory access (DMA) operations.
1. A method for denoising an image, the method comprising:
receiving an input image; and
performing a multiscale filtering process on the input image to generate a filtered image, wherein the multiscale filter process includes, for one or more resolution scales of a plurality of different resolution scales;
denoising an image at a different resolution scale of the plurality of different resolution scales to produce a denoised image;
downscaling the denoised image to produce a downscaled image;
upscaling the downscaled image to produce an upscaled image; and
fusing the upscaled imaged with the denoised image.
2. The method of claim 1 , wherein the multiscale filtering process includes, for a resolution scale of the plurality of different resolution scales: temporarily buffering the denoised image at a memory using direct memory access (DMA) operations.
3. The method of claim 2 , wherein each operation at a corresponding resolution scale of the plurality of different resolution scales is performed by corresponding separate functional block of a hardcoded hardware architecture of a system on a chip (SOC) coupled to the memory, the hardware architecture including at least one DMA engine.
4. The method of claim 1 , further comprising: displaying the filtered image.
5. A method for multiscale noise filtering of an image, the method comprising: receiving an input image to be filtered;
recursively performing, for each level of a plurality of levels in a first sequence, a denoising operation on an image received from a previous level in the first sequence followed by a downsampling operation on a resulting denoised image, followed by an upsampling operation on a resulting downsampled image, and then followed by a residue operation that subtracts a resulting upsampled image at the level from the denoised image at the level to generate a residue image for the level; and
recursively performing, for each level of the plurality of levels in a second sequence opposite of the first sequence, an upscaling operation to upscale a fused image of a previous level, followed by a fusion operation to combine a resulting upscaled fused image with the residue image of a next level in the second sequence to generate a fused image for the level; and
providing a resulting final fused image as a filtered image for at least one of storage, display, transmission, or encoding.
6. The method of claim 5 , wherein, for a level of the plurality of levels in the first sequence, the method includes:
receiving an output image from a previous level in the first sequence;
performing a denoising operation to generate a denoised image;
downsampling the denoised image to generate a downsampled image;
providing the downsampled image as an output image to the next level in the first sequence;
upsampling the downsampled image to generate an upsampled image; and
subtracting the upsampled image from the denoised image to generate a residue image for the level.
7. The method of claim 6 , wherein, for a level of the plurality of levels in the first sequence, the method further includes:
storing the denoised image of the level to a memory using at least one direct memory access operation;
storing the downsampled image of the level to the memory using at least one direct memory access operation;
accessing the downsampled image of the level from the memory using at least one DMA operation for use in upsampling the downsampled image to generate the upsampled image; and
accessing the denoised image for the level from the memory using at least one DMA operation for use in subtracting the upsampled image from the denoised image.
8. The method of claim 7 , wherein receiving the output image from a previous level comprises accessing the denoised image of the previous level from the memory using at least one DMA operation, the accessed denoised image being the output image from the previous level.
9. The method of claim 8 , wherein each operation performed at each level of the plurality of levels is performed by corresponding separate functional block of a hardcoded hardware architecture of a system on a chip (SOC) coupled to a memory, the hardware architecture including at least one DMA engine to perform DMA operations for the functional blocks of the hardcoded hardware architecture.
10. The method of claim 8 , wherein, for a level of the plurality of levels in the second sequence, the method includes:
upscaling a fused image from a previous level in the second sequence to generate an upscaled fused image; and
combining the upscaled fused image with the residue image of the level to generate a fused image for use at a next level in the second sequence.
11. The method of claim 6 , wherein, for a level of the plurality of levels in the second sequence, the method includes:
upscaling a fused image from a previous level in the second sequence to generate an upscaled fused image; and
combining the upscaled fused image with the residue image of the level to generate a fused image for use at a next level in the second sequence.
12. The method of claim 5 , wherein the input image is a chrominance image of an image.
13. An image processing system comprising:
an input to receive an input image; and
a multiscale filtering circuitry configured to perform a multiscale filtering process on the input image to generate a filtered image, wherein the multiscale filtering process includes, for one or more resolution scales of a plurality of different resolution scales;
denoising an image at a different resolution scale of the plurality of different resolution scales to produce a denoised image;
downscaling the denoised image to produce a downscaled image;
upscaling the downscaled image to produce an upscaled image; and
fusing the upscaled imaged with the denoised image.
14. The image processing system of claim 13 , further comprising:
a memory coupled to the multiscale filtering circuitry; and
wherein the multiscale filtering circuitry includes at least one direct memory access (DMA) engine to temporarily buffer, for a resolution scale of the plurality of different resolution scales, the denoised image at the memory.
15. The image processing system of claim 14 , wherein each operation at a corresponding resolution scale of the plurality of different resolution scales is performed by a corresponding separate functional block of a hardcoded hardware architecture of a system on a chip (SOC) coupled to the memory.
16. The image processing system of claim 13 , further comprising:
an imaging camera connected to the input and configured to capture an image of a scene, wherein the input image is a chrominance portion of the image captured by an imaging camera.
17. The image processing system of claim 13 , further comprising:
an output coupleable to a display for providing the filtered image for display.
18. An image processing system for multiscale filtering of an image, the image processing system comprising:
an input to receive an input image to be filtered; and
a system on a chip (SoC) having a hardware architecture implementing a set of hardware functional blocks configured to:
recursively perform, for each level of a plurality of levels in a first sequence, a denoising operation on an image received from a previous level in the first sequence followed by a downsampling operation on a resulting denoised signal, followed by an upsampling operation on a resulting downsampled image, and then followed by a residue operation that subtracts a resulting upsampled image at the level from the denoised image at the level to generate a residue image for the level; and
recursively perform, for each level of the plurality of levels in a second sequence opposite of the first sequence, an upscaling operation to upscale the denoised image of the level, followed by a fusion operation to combine a resulting upscaled image with the residue image of the next level in the second sequence; and
provide a resulting filtered image for at least one of storage, transmission, or encoding.
19. The image processing system of claim 18 , wherein, for a level of the plurality of levels in the first sequence, the set of hardware functional blocks is configured to:
receive an output image from a previous level in the first sequence;
perform a denoising operation to generate a denoised image;
downsample the denoised image to generate a downsampled image;
provide the downsampled image as an output image to the next level in the first sequence;
upsample the downsampled image to generate an upsampled image; and
subtract the upsampled image from the denoised image to generate a residue image for the level.
20. The image processing system of claim 19 , wherein, for a level of the plurality of levels in the first sequence, the set of hardware functional blocks is configured to:
store the denoised image of the level to a memory using at least one direct memory access (DMA) operation by a DMA engine;
store the downsampled image of the level to the memory using at least one direct memory access operation by a DMA engine;
access the downsampled image of the level from the memory using at least one DMA operation by a DMA engine for use in upsampling the downsampled image to generate the upsampled image; and
access the denoised image for the level from the memory using at least one DMA operation by a DMA engine for use in subtracting the upsampled image from the denoised image.
21. The image processing system of claim 20 , wherein the set of hardware functional blocks is configured to receive the output image from a previous level by accessing the denoised image of the previous level from the memory using at least one DMA operation of a DMA engine, the accessed denoised image being the output image from the previous level.
22. The image processing system of claim 19 , wherein, for a level of the plurality of levels in the second sequence, the set of hardware functional blocks is configured to:
upscale a fused image from a previous level in the second sequence to generate an upscaled fused image; and
combine the upscaled fused image with the residue image of the level to generate a fused image for use at a next level in the second sequence.
23. The image processing system of claim 18 , further comprising:
an imaging camera connected to the input and configured to capture an image of a scene, wherein the input image is a chrominance portion of the image captured by an imaging camera.