Adaptive deep learning model for noisy image super-resolution
Embodiments described herein are generally directed to an end-to-end trainable degradation restoration network (DRN) that enhances the ability of a super-resolution (SR) subnetwork to deal with noisy low-resolution images. An embodiment of a method includes estimating, by a noise estimator (NE) subnetwork of the DRN, an estimated noise map for a noisy input image; and predicting, by the SR subnetwork of the DRN, a clean upscaled image based on the input image and the noise map by, for each of multiple conditional residual dense blocks (CRDBs) stacked within one or more cascade blocks representing the SR subnetwork, adjusting, by a noise control layer of the CRDB that follows a stacked set of a multiple residual dense blocks of the CRDB, feature values of an intermediate feature map associated with the input image by applying (i) a scaling factor and (ii) an offset factor derived from the noise map.
1. A method for performing image super-resolution comprising:
estimating, by a noise estimator (NE) convolutional neural network (CNN) subnetwork of a degradation restoration network (DRN), an estimated noise map for a noisy input image; and
predicting, by a super-resolution (SR) CNN subnetwork of the DRN, a super-resolved clean image based on the noisy input image and the estimated noise map by adjusting feature values of an intermediate feature map associated with the noisy input image including applying to the feature values (i) a scaling factor determined during one or more training phases of the DRN and (ii) an offset factor derived from the estimated noise map, wherein the NE CNN subnetwork learned to estimate the noise map and the SR CNN subnetwork learned to predict super-resolved clean images concurrently during the one or more training phases.
2. The method of claim 1 , wherein the SR CNN subnetwork comprises one or more cascade blocks made up of a plurality of conditional residual dense blocks (CRDBs), each including a residual dense block (RDB) and a noise control layer and wherein said adjusting is performed by the noise control layer of each of the plurality of CRDBs.
3. The method of claim 1 , wherein the one or more training phases include a first training phase during which the SR CNN subnetwork is trained to upscale training images and the NE CNN subnetwork is trained to estimate a noise map associated with the training images.
4. The method of claim 3 , wherein the training images comprise synthetic noisy images.
5. The method of claim 3 , further comprising determining by each noise shifting (NS) CNN subnetwork of one or more NS CNN subnetworks of the DRN, an additive noise map to be combined with the estimated noise map input to a corresponding cascade block of the one or more cascade blocks.
6. The method of claim 5 , further comprising after the first training phase is complete:
adding the one or more NS CNN subnetworks to the DRN; and
performing a second training phase of the one or more training phases, during which each of the one or more NS CNN subnetworks independently learn respective additive noise maps for the corresponding cascade block based on the training images.
7. An apparatus for performing image super-resolution comprising:
one or more processors, including a graphics processor; and
instructions which when executed by the one or more processors cause the one or more processor to:
estimate, by a noise estimator (NE) convolutional neural network (CNN) subnetwork of a degradation restoration network (DRN), an estimated noise map for a noisy input image; and
predict, by a super-resolution (SR) CNN subnetwork of the DRN, a super-resolved clean image based on the noisy input image and the estimated noise map by adjusting feature values of an intermediate feature map associated with the noisy input image including applying to the feature values (i) a scaling factor determined during one or more training phases of the DRN and (ii) an offset factor derived from the estimated noise map, wherein the NE CNN subnetwork learned to estimate the noise map and the SR CNN subnetwork learned to predict super-resolved clean images concurrently during the one or more training phases.
8. The apparatus of claim 7 , wherein the SR CNN subnetwork comprises one or more cascade blocks made up of a plurality of conditional residual dense blocks (CRDBs), each including a residual dense block (RDB) and a noise control layer and wherein said adjusting is performed by the noise control layer of each of the plurality of CRDBs.
9. The apparatus of claim 7 , wherein the one or more training phases include a first training phase during which the SR CNN subnetwork is trained to upscale training images and the NE CNN subnetwork is trained to estimate a noise map associated with the training images.
10. The apparatus of claim 9 , wherein the training images comprise synthetic noisy images.
11. The apparatus of claim 9 , wherein the instructions further cause the one or more processors to determine by each noise shifting (NS) CNN subnetwork of one or more NS CNN subnetworks of the DRN, an additive noise map to be combined with the estimated noise map input to a corresponding cascade block of the one or more cascade blocks.
12. The apparatus of claim 11 , wherein the instructions further cause the one or more processors to, after the first training phase is complete:
add the one or more NS CNN subnetworks to the DRN; and
perform a second training phase of the one or more training phases, during which each of the one or more NS CNN subnetworks independently learn respective additive noise maps for the corresponding cascade block based on the training images.
13. The apparatus of claim 12 , wherein the instructions further cause the one or more processor to,, prior to performance of the second training phase, freeze a training state the NE CNN subnetwork and the SR CNN subnetwork.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the one or more training phases include a first training phase during which the SR CNN subnetwork is trained to upscale training images and the NE CNN subnetwork is trained to estimate a noise map associated with the training images.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the training images comprise synthetic noisy images.
16. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further cause the one or more processors to determine by each noise shifting (NS) CNN subnetwork of one or more NS CNN subnetworks of the DRN, an additive noise map to be combined with the estimated noise map input to a corresponding cascade block of the one or more cascade blocks.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further cause the one or more processors to, after the first training phase is complete:
add the one or more NS CNN subnetworks to the DRN; and
perform a second training phase of the one or more training phases, during which each of the one or more NS CNN subnetworks independently learn respective additive noise maps for the corresponding cascade block based on the training images.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the instructions further cause the one or more processor to, prior to performance of the second training phase, freeze a training state the NE CNN subnetwork and the SR CNN subnetwork.
19. A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processors, including a graphics processor, causes the one or more processors to:
estimate, by a noise estimator (NE) convolutional neural network (CNN) subnetwork of a degradation restoration network (DRN), an estimated noise map for a noisy input image; and
predict, by a super-resolution (SR) CNN subnetwork of the DRN, a super-resolved clean image based on the noisy input image and the estimated noise map by adjusting feature values of an intermediate feature map associated with the noisy input image including applying to the feature values (i) a scaling factor determined during one or more training phases of the DRN and (ii) an offset factor derived from the estimated noise map, wherein the NE CNN subnetwork learned to estimate the noise map and the SR CNN subnetwork learned to predict super-resolved clean images concurrently during the one or more training phases.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the SR CNN subnetwork comprises one or more cascade blocks made up of a plurality of conditional residual dense blocks (CRDBs), each including a residual dense block (RDB) and a noise control layer and wherein said adjusting is performed by the noise control layer of each of the plurality of CRDBs.