Methods and apparatus for end-to-end unsupervised multi-document blind image denoising
Methods and apparatus for end-to-end unsupervised multi-document blind image denoising is presented. The multi-document blind image denoiser removes various noise types from noisy documents without paired target cleaned documents and preserves the contents for optical character recognition. The end-to-end unsupervised multi-document blind image denoiser integrates a Mixture of Experts with a cycle-consistent GAN as the base network that effectively removes multiple types of noise, including salt & pepper noise, blurred and/or faded text, as well as watermarks from documents at various levels of intensity.
1 . An apparatus comprising:
a processor; and
a memory operatively coupled to the processor, the memory storing instructions to cause the processor to:
receive a noisy image,
generate a denoised image based on the noisy image,
generate, based on the denoised image, a noised image and a classification of the denoised image,
generate, based on the noised image, a classification of the noised image, and
generate a clean image based on a comparison between the classification of the denoised image and the classification of the noised image.
2 . The apparatus of claim 1 , wherein:
the noisy image is associated with a document type; and
the classification of the denoised image and the classification of the noised image indicate the document type.
3 . The apparatus of claim 1 , wherein:
the noisy image is associated with at least one of a structured document type, a semi-structured document type, or an unsupervised document type; and
the classification of the denoised image and the classification of the noised image indicate the at least one of the structured document type, the semi-structured document type, or the unsupervised document type.
4 . The apparatus of claim 1 , wherein the memory stores instructions to further cause the processor to:
generate, based on the denoised image, a first noise intensity value; and
generate, based on the noised image, a second noise intensity value, the clean image being generated based further on a comparison between the first noise intensity value and the second noise intensity value.
5 . The apparatus of claim 1 , wherein the memory stores instructions to further cause the processor to:
determine, based on the denoised image, a first noise type; and
determine, based on the noised image, a second noise type, the clean image being generated based further on a comparison between the first noise type and the second noise type.
6 . The apparatus of claim 1 , wherein the memory stores instructions to further cause the processor to:
classify the denoised image as having at least one of salt & pepper (S&P) noise, blurred text, faded text, a shading, or a watermark; and
classify the noised image as having the at least one of the S&P noise, the blurred text, the faded text, the shading, or the watermark, the clean image being generated based further on the denoised image and the noised image being classified as having the at least one of the S&P noise, the blurred text, the faded text, the shading, or the watermark.
7 . The apparatus of claim 1 , wherein:
the clean image is a clean instance of the noisy image.
8 . The apparatus of claim 1 , wherein:
the instructions to cause the processor to generate the denoised image include instructions to cause the processor to:
compute, via an embedder and based on the noisy image, a denoising mixture weight for each denoising gate from a plurality of denoising gates associated with a plurality of channels of the noisy image, to generate a plurality of denoising mixture weights, and
apply the plurality of denoising mixture weights to the noisy image to generate the denoised image; and
the instructions to cause the processor to generate the noised image include instructions to cause the processor to:
compute, via the embedder and based on the denoised image, a noising mixture weight for each noising gate from a plurality of noising gates associated with a plurality of channels of the denoised image, to generate a plurality of noising mixture weights, and
apply the plurality of noising mixture weights to the denoised image to generate the noised image.
9 . An apparatus comprising:
a processor; and
a memory operatively coupled to the processor, the memory storing instructions to cause the processor to:
receive a first image;
input the first image to a first machine learning model to generate (i) a second image that depicts more noise than the first image and (ii) a classification of the first image;
input the second image to a second machine learning model to generate a classification of the second image; and
generate a third image based on a comparison between the classification of the first image and the classification of the second image.
10 . The apparatus of claim 9 , wherein the memory stores instructions to further cause the processor to:
receive, before receiving the first image, a fourth image that depicts less noise than the first image; and
input the fourth image to a third machine learning model to generate the first image.
11 . The apparatus of claim 9 , wherein the first image is a denoised image, and the memory stores instructions to further cause the processor to:
receive, before receiving the denoised image, a noisy image; and
input the noisy image to a third machine learning model to generate the denoised image.
12 . The apparatus of claim 9 , wherein the classification of the first image and the classification of the second image indicate a document type represented by the first image and the second image.
13 . The apparatus of claim 9 , wherein:
the first image and the second image are associated with at least one of a structured document type, a semi-structured document type, or an unsupervised document type; and
the classification of the first image and the classification of the second image indicate the at least one of the structured document type, the semi-structured document type, or the unsupervised document type.
14 . The apparatus of claim 9 , wherein:
the first image and the second image are associated with a noise type; and
the classification of the first image and the classification of the second image indicate the noise type.
15 . The apparatus of claim 9 , wherein:
the first image and the second image depict at least one of salt & pepper (S&P) noise, blurred text, faded text, a shading, or a watermark; and
the classification of the first image and the classification of the second image indicate the at least one of the S&P noise, the blurred text, the faded text, the shading, or the watermark.
16 . The apparatus of claim 9 , wherein the memory stores instructions to further cause the processor to:
generate, based on the first image, a first noise intensity value; and
generate, based on the second image, a second noise intensity value, the third image being generated based further on a comparison between the first noise intensity value and the second noise intensity value.
17 . The apparatus of claim 9 , wherein:
a convolutional neural network (CNN) includes at least one of the first machine learning model or the second machine learning model.
18 . The apparatus of claim 9 , wherein the instructions to cause the processor to generate the second image include instructions to cause the processor to:
compute, via a third machine learning model and based on the first image, a mixture weight for each gate from a plurality of gates associated with a plurality of channels of the first machine learning model, to generate a plurality of mixture weights; and
generate the second image based on the plurality of mixture weights.
19 . A method, comprising:
receiving, via a processor, a noisy image without a clean image;
generating, via the processor, a denoised image based on the noisy image;
generating, via the processor and based on the denoised image, a noised image and a classification of the denoised image,
generating, via the processor and based on the noised image, a classification of the noised image, and
generating, via the processor, the clean image based on a comparison between the classification of the denoised image and the classification of the noised image.
20 . The method of claim 19 , wherein:
the noisy image is associated with a document type; and
the classification of the denoised image and the classification of the noised image indicate the document type.