Model based document image enhancement
Systems and methods are disclosed for model based document image enhancement. Instead of requiring paired dirty and clean images for training a model to clean document images (which may cause privacy concerns), two models are trained on the unpaired images such that only the dirty images are accessed or only the clean images are accessed at one time. One model is a first implicit model to translate the dirty images from a source space to a latent space, and the other model is a second implicit model to translate the images from the latent space to clean images in a target space. The second implicit model is trained based on translating electronic document images in the target space to the latent space. In some implementations, the implicit models are diffusion models, such as denoising diffusion implicit models based on solving ordinary differential equations.
1. A computer-implemented method for document image enhancement, the method comprising:
obtaining an electronic document image by a machine learning (ML) model, wherein the electronic document image is generated from scanning a physical document; and
enhancing the electronic document image by the ML model, including:
translating the electronic document image in a source space to a latent space by a first implicit probabilistic model of the ML model, wherein the first implicit probabilistic model is trained based on translating electronic document images in the source space to the latent space; and
translating the electronic document image in the latent space to a target space by a second implicit probabilistic model of the ML model, wherein:
the second implicit probabilistic model is trained independently from the first implicit probabilistic model;
the second implicit probabilistic model is trained based on translating electronic document images in the target space to the latent space; and
the second implicit probabilistic model includes a second cycle consistent model to translate the electronic document image between the latent space and the target space, wherein the second cycle consistent model is configured to reverse the function of translating a target space document image to a latent space document image to translate the electronic document image in the latent space to the target space; and
providing the enhanced electronic document image in the target space to an object character recognition (OCR) engine to perform OCR.
2. The computer-implemented method of claim 1 , wherein:
the first implicit probabilistic model includes a first cycle consistent model to translate the electronic document image between the source space and the latent space.
3. The computer-implemented method of claim 2 , wherein:
the first cycle consistent model includes a first diffusion model; and
the second cycle consistent model includes a second diffusion model.
4. The computer-implemented method of claim 3 , wherein the first diffusion model includes a first denoising diffusion implicit model (DDIM) based on solving a first ordinary differential equation (ODE) for encoding the electronic document image from the source space to the latent space.
5. The computer-implemented method of claim 3 , wherein the second diffusion model includes a second denoising diffusion implicit model (DDIM) based on solving a second ordinary differential equation (ODE) for encoding the electronic document image from the target space to the latent space.
6. The computer-implemented method of claim 1 , further comprising performing OCR on the enhanced electronic document image in the target space by the OCR engine to generate an OCR document.
7. A computing system for document image enhancement, the computing system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, causes the computing system to perform operations comprising:
obtaining an electronic document image by a machine learning (ML) model of the computing system, wherein the electronic document image is generated from scanning a physical document;
enhancing the electronic document image by the ML model, including:
translating the electronic document image in a source space to a latent space by a first implicit probabilistic model of the ML model, wherein the first implicit probabilistic model is trained based on translating electronic document images in the source space to the latent space; and
translating the electronic document image in the latent space to a target space by a second implicit probabilistic model of the ML model, wherein:
the second implicit probabilistic model is trained independently from the first implicit probabilistic model;
the second implicit probabilistic model is trained based on translating electronic document images in the target space to the latent space; and
the second implicit probabilistic model includes a second cycle consistent model to translate the electronic document image between the latent space and the target space, wherein the second cycle consistent model is configured to reverse the function of translating a target space document image to a latent space document image to translate the electronic document image in the latent space to the target space; and
providing the enhanced electronic document image in the target space to an object character recognition (OCR) engine to perform OCR.
8. The computing system of claim 7 , wherein:
the first implicit probabilistic model includes a first cycle consistent model to translate the electronic document image between the source space and the latent space.
9. The computing system of claim 8 , wherein:
the first cycle consistent model includes a first diffusion model; and
the second cycle consistent model includes a second diffusion model.
10. The computing system of claim 9 , wherein the first cycle consistent model includes a first denoising diffusion implicit model (DDIM) based on solving a first ordinary differential equation (ODE) for encoding the electronic document image from the source space to the latent space.
11. The computing system of claim 9 , wherein the second cycle consistent model includes a second denoising diffusion implicit model (DDIM) based on solving a second ordinary differential equation (ODE) for encoding the electronic document image from the target space to the latent space.
12. The computing system of claim 7 , wherein the operations further comprise performing OCR on the enhanced electronic document image in the target space by the OCR engine to generate an OCR document.