Image-to-image mapping by iterative de-noising
A method includes receiving training data comprising a plurality of pairs of images. Each pair comprises a noisy image and a denoised version of the noisy image. The method also includes training a multi-task diffusion model to perform a plurality of image-to-image translation tasks, wherein the training comprises iteratively generating a forward diffusion process by predicting, at each iteration in a sequence of iterations and based on a current noisy estimate of the denoised version of the noisy image, noise data for a next noisy estimate of the denoised version of the noisy image, updating, at each iteration, the current noisy estimate to the next noisy estimate by combining the current noisy estimate with the predicted noise data, and determining a reverse diffusion process by inverting the forward diffusion process to predict the denoised version of the noisy image. The method additionally includes providing the trained diffusion model.
1. A computer-implemented method, comprising:
receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises a noisy image and a denoised version of the noisy image;
training, based on the training data, a multi-task diffusion model to perform a plurality of image-to-image translation tasks, wherein the training comprises:
iteratively generating a forward diffusion process by predicting, at each iteration in a sequence of iterations and based on a current noisy estimate of the denoised version of the noisy image, noise data for a next noisy estimate of the denoised version of the noisy image,
updating, at each iteration, the current noisy estimate to the next noisy estimate by combining the current noisy estimate with the predicted noise data, and
determining a reverse diffusion process by inverting the forward diffusion process to predict the denoised version of the noisy image; and
providing, by the computing device, the trained multi-task diffusion model.
2. The method of claim 1 , further comprising:
sampling, at a first iteration of the sequence of iterations, an initial noise data from a predefined noise distribution.
3. The method of claim 2 , wherein the predefined noise distribution is a standard Normal distribution.
4. The method of claim 1 , wherein each iteration in the sequence of iterations is associated with a respective noise level parameter, and wherein the predicting of the noise data at each iteration is based on the noise level parameter associated with the iteration.
5. The method of claim 4 , wherein for each iteration in the sequence of iterations, the updating of the current noisy estimate to the next noisy estimate is performed by combining the predicted noise data with the current estimate in accordance with the noise level parameter associated with the iteration.
6. The method of claim 4 , wherein for each iteration prior to the final iteration in the sequence of iterations, the updating of the current estimate comprises:
sampling additional noise data from a predefined noise distribution; and
updating the current estimate based on: (i) the additional noise data, and (iii) the noise level parameter associated with the iteration.
7. The method of claim 1 , wherein the predicting of the noise data comprises:
estimating actual noise in the noisy image based on the corresponding denoised version of the noisy image.
8. The method of claim 7 , wherein the diffusion model is a neural network, and the training of the neural network further comprising:
updating one or more current values of a set of parameters of the neural network using one or more gradients of an objective function that measures an error between: (i) the predicted noise data, and (ii) the actual noise data in the noisy target output image.
9. The method of claim 8 , wherein the error is one of an L 1 error or an L 2 error.
10. The method of claim 1 , wherein the plurality of pairs of images in the training data correspond to each of the plurality of image-to-image translation tasks.
11. The method of claim 1 , wherein the plurality of image-to-image translation tasks comprise one or more of a colorization task, an uncropping task, an inpainting task, a decompression artifact removal task, a super-resolution task, a de-noising task, or a panoramic image generation task.
12. The method of claim 1 , wherein the diffusion model is a neural network comprising one or more self-attention refinement neural network layers.
13. A computer-implemented method, comprising:
receiving, by a computing device, an input image;
applying a multi-task diffusion model to predict a denoised version of the input image by applying a reverse diffusion process, the diffusion model having been trained on a plurality of pairs of images, wherein each pair comprises a noisy image and a denoised version of the noisy image, and the diffusion model having been trained to perform a plurality of image-to-image translation tasks, the training comprising:
iteratively generating a forward diffusion process by predicting, at each iteration in a sequence of iterations and based on a current noisy estimate of the denoised version of the noisy image, noise data for a next noisy estimate of the denoised version of the noisy image,
updating, at each iteration, the current noisy estimate to the next noisy estimate by combining the current noisy estimate with the predicted noise data, and
determining the reverse diffusion process by inverting the forward diffusion process to predict the denoised version of the input image; and
providing, by the computing device, the predicted denoised version of the input image.
14. The method of claim 13 , wherein the plurality of image-to-image translation tasks comprise one or more of a colorization task, an uncropping task, an inpainting task, a decompression artifact removal task, a super-resolution task, a de-noising task, or a panoramic image generation task.
15. The method of claim 13 , wherein the input image comprises one or more missing interior regions, and the predicted denoised version comprises an inpainting of the one or more missing interior regions.
16. The method of claim 13 , wherein the input image comprises one or more blurred portions, and the predicted denoised version is a deblurred version of the input image.
17. The method of claim 13 , wherein the input image is a cropped image, and the predicted denoised version is an uncropped version of the input image.
18. The method of claim 13 , wherein the input image comprises one or more decompression artifacts, and the applying of the multi-task diffusion model comprises removing the one or more decompression artifacts.
19. The method of claim 13 , wherein the input image comprises one or more image distortions, and the applying of the multi-task diffusion model comprises correcting the one or more image distortions.
20. A computer-implemented method, comprising:
receiving, by a computing device, a first input image comprising a first image degradation and a second input image comprising a second image degradation;
applying a multi-task diffusion model to predict respective denoised versions of the first input image and the second input image by applying a reverse diffusion process, wherein the predicting involves removing the first image degradation from the first input image and the second image degradation from the second input image, and the diffusion model having been trained to:
iteratively generate a forward diffusion process, and
determine the reverse diffusion process by inverting the forward diffusion process to predict the respective denoised versions of the first input image and the second input image; and
providing, by the computing device, the respective denoised versions of the first input image and the second input image.
21. The method of claim 20 , wherein:
the first input image comprises one or more missing interior regions, and the predicted denoised version of the first input image comprises an inpainting of the one or more missing interior regions, and
the second input image comprises one or more blurred portions, and the predicted denoised version is a deblurred version of the input image is a deblurred version of the second input image.
22. The method of claim 20 , wherein:
the first input image is a cropped image, and the predicted denoised version of the first input image is an uncropped version of the first input image, and
the second input image comprises one or more blurred portions, and the predicted denoised version is a deblurred version of the input image is a deblurred version of the second input image.
23. The method of claim 20 , wherein:
the first input image is a cropped image, and the predicted denoised version of the first input image is an uncropped version of the first input image, and
the second input image comprises one or more decompression artifacts, and the applying of the multi-task diffusion model comprises removing the one or more decompression artifacts.
24. The method of claim 20 , wherein:
the first input image comprises one or more decompression artifacts, and the applying of the multi-task diffusion model comprises removing the one or more decompression artifacts, and
the second input image comprises one or more image distortions, and the applying of the multi-task diffusion model comprises correcting the one or more image distortions.
25. The method of claim 20 , wherein:
the first input image comprises one or more missing interior regions, and the predicted denoised version of the first input image comprises an inpainting of the one or more missing interior regions, and
the second input image comprises one or more image distortions, and the applying of the multi-task diffusion model comprises correcting the one or more image distortions.
26. The method of claim 20 , wherein:
the first input image comprises one or more missing interior regions, and the predicted denoised version of the first input image comprises an inpainting of the one or more missing interior regions, and
the second input image is a grayscale image, and the predicted denoised version of the second input image comprises a colorization of the grayscale image.