System and method for training of noise model using noisy signal pairs
A noise model is iteratively trained to simulate introduction of noise by a capture device, by use of a denoiser and a training data set of pairs of noisy signals. First and second noisy signals of each pair are independently sampled by the capture device from source information corresponding to the pair. Each iteration of training obtains first and second denoised signals from respective noisy signals, then optimizes at least one loss function which sums first and second terms to train both the noise model and the denoiser, where the first term is based on the first denoised signal and the second noisy signal, and the second term is based on the second denoised signal and the first noisy signal. By using noisy samples, the complexities of obtaining “clean” signals are avoided. By using “cross-sample” loss functions, convergence on undesired training results is avoided without complex regularization.
1. A method for training of a noise model to simulate introduction of noise by a capture device, the method comprising:
establishing the noise model and a denoiser;
obtaining a training data set comprising at least one pair of noisy signals, the pair of noisy signals comprising a first noisy signal and a second noisy signal, both the first and the second noisy signals being independently sampled by the capture device from source information corresponding to the pair of noisy signals;
iteratively training, by at least one processor, the noise model, each iteration of training comprising:
obtaining a first denoised signal by denoising the first noisy signal using the denoiser,
obtaining a second denoised signal by denoising the second noisy signal using the denoiser,
training the noise model and the denoiser each based on optimization of at least one loss function, each loss function summing at least a first term and a second term, the first term of each loss function being based on the first denoised signal and the second noisy signal, the second term of each loss function being based on the second denoised signal and the first noisy signal; and
outputting the trained noise model as a trained noise model for the capture device.
2. The method of claim 1 , wherein the training data set comprises a plurality of pairs of noisy signals, each pair of noisy signals used as the first and the second noisy signals in at least one iteration of training.
3. The method of claim 1 , further comprising determining a first noise estimate based on a difference between the first noisy signal and the second denoised signal, and determining a second noise estimate is determined based on a difference between the second noisy signal and the first denoised signal,
wherein the first term of a first loss function of the at least one loss function is based on the first denoised signal and the second noise estimate, and the second term of the first loss function is based on the second denoised signal and the first noise estimate.
4. The method of claim 3 , wherein the first loss function comprises a loss function of the noise model, a second loss function of the at least one loss function comprises a loss function of the denoiser, the loss function of the noise model is based on the first and the second noise estimates and the first and the second denoised signals, and the loss function of the denoiser is based on the first and the second noisy signals and the first and the second denoised signals.
5. The method of claim 4 , wherein the first term of the loss function of the noise model is based on a probability of the noise model simulating introduction of the first noise estimate to the second denoised signal, and the second term of the loss function of the noise model is based on a probability of the noise model simulating introduction of the second noise estimate to the first denoised signal.
6. The method of claim 1 , wherein the at least one loss function comprises a loss function of the noise model, the first term of the loss function of the noise model is based on a probability of the noise model generating the first noisy signal from the second denoised signal, and the second term of the loss function of the noise model is based on a probability of the noise model generating the second noisy signal from the first denoised signal.
7. The method of claim 6 , wherein the loss function of the noise model comprises a negative log-likelihood function.
8. The method of claim 1 , wherein the at least one loss function comprises a loss function of the denoiser, the first term of the loss function of the denoiser is based on a difference between the first noisy signal and the second denoised signal, and the second term of the loss function of the denoiser is based on a difference between the second noisy signal and the first denoised signal.
9. The method of claim 8 , wherein the loss function of the denoiser comprises a mean squared error function.
10. The method of claim 1 , wherein the noise model and the denoiser are concurrently trained by backpropagation.
11. The method of claim 1 , wherein the training data set comprises a plurality of pairs of noisy signals, each iteration of the training further comprising selecting one of the plurality of pairs of noisy signals for use as the first and the second noisy signals.
12. The method of claim 11 , wherein the selecting of one of the plurality of pairs of noisy signals is based on a presently selected subset of the plurality of pairs of noisy signals.
13. The method of claim 1 , wherein the denoiser comprises a neural network.
14. The method of claim 1 , wherein the noise model comprises a Noise Flow model.
15. The method of claim 1 , wherein the capture device comprises an image sensor, and the first and the second noisy signal each comprise image data.
16. The method of claim 1 , further comprising outputting the trained denoiser as a trained denoiser for the capture device.
17. The method of claim 1 , further comprising simulating the introduction of noise by the capture device to a signal based on the noise model.
18. A signal denoising method for signals captured by a capture device, the method comprising:
training a noise model to simulate introduction of noise by the capture device, using the method of claim 1 ; and
denoising a noisy signal captured by the capture device based on parameters of the noise model.
19. A system for training of a noise model to simulate introduction of noise by a capture device, the system comprising:
at least one non-volatile memory electrically configured to store instructions, the noise model, a denoiser, and a training data set, the training data set comprising at least one pair of noisy signals, the pair of noisy signals comprising a first noisy signal and a second noisy signal, both the first and the second noisy signals being independently sampled by the capture device from source information corresponding to the pair of noisy signals; and
at least one processor operatively connected to the at least one non-volatile memory, the at least one processor being configured to execute the instructions to:
iteratively train the noise model, each iteration of training comprising:
obtaining a first denoised signal by denoising the first noisy signal using the denoiser,
obtaining a second denoised signal by denoising the second noisy signal using the denoiser,
training the noise model and the denoiser each based on optimization of at least one loss function, each loss function summing at least a first term and a second term, the first term of each loss function being based on the first denoised signal and the second noisy signal, the second term of each loss function being based on the second denoised signal and the first noisy signal; and
output the trained noise model as a trained noise model for the capture device.
20. A non-transitory computer-readable recording medium having recorded thereon instructions executable by at least one processor to perform a method for training of a noise model to simulate introduction of noise by a capture device, the method comprising:
obtaining a training data set comprising at least one pair of noisy signals, the pair of noisy signals comprising a first noisy signal and a second noisy signal, both the first and the second noisy signals being independently sampled by the capture device from source information corresponding to the pair of noisy signals;
iteratively training the noise model and a denoiser, each iteration of training comprising:
obtaining a first denoised signal by denoising the first noisy signal using the denoiser,
obtaining a second denoised signal by denoising the second noisy signal using the denoiser,
training the noise model and the denoiser each based on optimization of at least one loss function, each loss function summing at least a first term and a second term, the first term of each loss function being based on the first denoised signal and the second noisy signal, the second term of each loss function being based on the second denoised signal and the first noisy signal; and
outputting the trained noise model as a trained noise model for the capture device.