Image super-resolution method using frequency domain features
An image super-resolution method includes performing frequency domain transformation on a first image to obtain a spectral feature of the first image, the spectral feature representing a distribution of a grayscale gradient in the first image. The method further includes performing blur kernel prediction based on the spectral feature to obtain a blur kernel of the first image, the blur kernel being a convolution kernel. The method also includes performing super-resolution processing on the first image based on the blur kernel to generate a super-resolved image, a definition of the super-resolved image being higher than a definition of the first image.
1 . An image super-resolution method, comprising:
obtaining one or more second images based on a first image, each of the one or more second images having a first size;
performing frequency domain transformation on the one or more second images to obtain one or more first spectrograms of the one or more second images;
performing down-sampling processing on each of the one or more first spectrograms to obtain one or more second spectrograms of the one or more second images, the one or more second spectrograms corresponding to a spectral feature of the first image, the spectral feature representing a distribution of a grayscale gradient in the first image, and each of the one or more second spectrograms having a second size less than the first size;
obtaining one or more blur kernels of the one or more second images based on application of a kernel prediction model to each of the one or more second spectrograms of the one or more second images, each of the one or more blur kernels being a convolution kernel;
generating one or more sub-super-resolved images based on the one or more second images and the one or more blur kernels; and
generating a super-resolved image based on the one or more sub-super-resolved images, a definition of the super-resolved image being higher than a definition of the first image.
2 . The method according to claim 1 , wherein
the obtaining the one or more second images includes:
based on a determination that an image size of the first image is greater than the first size, performing image cropping on the first image based on the first size to obtain at least two second images; and
the generating the super-resolved image includes:
performing pixel averaging processing on an intersection area of at least two sub-super-resolved images corresponding to the at least two second images; and
fusing the at least two sub-super-resolved images based on the processed intersection area to generate the super-resolved image.
3 . The method according to claim 1 , wherein
the obtaining the one or more second images includes:
based on a determination that an image size of the first image is less than the first size, performing image expansion on the first image based on the first size to obtain a single second image, the image expansion including copying and stitching processing of the first image; and
the generating the super-resolved image includes:
performing image cropping on a single sub-super-resolved image corresponding to the single second image to generate the super-resolved image.
4 . The method according to claim 1 , wherein
the performing the frequency domain transformation includes:
performing discrete fast Fourier transform on the one or more second images to obtain the one or more first spectrograms.
5 . The method according to claim 1 , wherein
the kernel prediction model includes at least one convolution layer and at least one transposed convolution layer, a number of the at least one convolution layer being consistent with a number of the at least one transposed convolution layer; and
the obtaining the one or more blur kernels includes, for each of the one or more blur kernels based on the respective one of the one or more second spectrograms:
performing convolution processing on the corresponding second spectrogram through the at least one convolution layer in the kernel prediction model to obtain a feature vector of the corresponding second spectrogram; and
performing transposed convolution processing on the feature vector through the at least one transposed convolution layer to output the corresponding blur kernel.
6 . The method according to claim 1 , wherein
the one or more sub-super-resolved images are generated based on a non-blind super-resolution model according to a spatial feature transform for multiple degradations (SFTMD) network or a residual channel attention network (RCAN).
7 . An image super-resolution method, comprising:
performing frequency domain transformation on an i-th sample image to obtain an i-th sample spectrogram of the i-th sample image, the i-th sample image being obtained based on an i-th real blur kernel, i being a positive integer;
inputting the i-th sample spectrogram into a kernel prediction model to obtain an i-th predicted blur kernel outputted by the kernel prediction model, the i-th predicted blur kernel being a convolution kernel;
obtaining i-th first data based on a combination of the i-th sample spectrogram and the i-th real blur kernel;
obtaining i-th second data based on a combination of the i-th sample spectrogram and the i-th predicted blur kernel; and
performing iterative training on the kernel prediction model, the iterative training including an i-th-training of the kernel prediction model based on application of a discriminator to the i-th first data and the i-th second data.
8 . The method according to claim 7 , wherein the performing the iterative training on the kernel prediction model comprises:
obtaining an i-th discrimination result from the discriminator based on the i-th first data and the i-th second data;
calculating an i-th model loss based on the i-th discrimination result; and
updating one or more parameters of the discriminator and one or more parameters of the kernel prediction model based on the i-th model loss.
9 . The method according to claim 7 , wherein
the obtaining the i-th first data is based on stitching of the i-th sample spectrogram and the i-th real blur kernel, and
the obtaining the i-th second data is based on stitching of the i-th sample spectrogram and the i-th predicted blur kernel.
10 . The method according to claim 8 , wherein
a loss function of the kernel prediction model and the discriminator is a cross-entropy loss function; and
the calculating the i-th model loss includes:
when a model parameter of the kernel prediction model in the cross-entropy loss function is kept constant, calculating an i-th model loss of the discriminator based on the i-th discrimination result of the discriminator, the i-th model loss of the discriminator being greater than an (i−1)-th model loss of the discriminator; and
when a model parameter of the discriminator in the cross-entropy loss function is kept constant, calculating an i-th model loss of the kernel prediction model based on the i-th discrimination result of the discriminator, the i-th model loss of the kernel prediction model being less than an (i−1)-th model loss of the kernel prediction model.
11 . The method according to claim 7 , further comprising:
performing image preprocessing on a first image to obtain the i-th sample image of a first size, wherein
the performing the frequency domain transformation includes:
performing discrete fast Fourier transform on the i-th sample image to obtain i-th first sample spectrogram of the i-th sample image; and
performing down-sampling processing on the i-th first sample spectrogram to obtain i-th second sample spectrogram of a second size, the second size being less than the first size; and
the inputting the i-th sample spectrogram into the kernel prediction model includes:
inputting the i-th second sample spectrogram as the i-th sample spectrogram into the kernel prediction model to obtain the i-th predicted blur kernel outputted by the kernel prediction model.
12 . The method according to claim 11 , wherein the performing the image preprocessing on the first image comprises:
based on a determination that an image size of the first image is greater than the first size, performing image cropping on the first image based on the first size to obtain the i-th sample image; and
based on a determination that the image size of the first image is less than the first size, performing image expansion on the first image based on the first size to obtain the i-th sample image, the image expansion including copying and stitching processing on the first image.
13 . The method according to claim 7 , wherein the method further comprises:
obtaining a third image, a definition of the third image being higher than a definition of a first image;
performing blur processing on the third image based on the i-th real blur kernel to obtain the first image; and
obtaining the i-th sample image based on the first image.
14 . An image super-resolution apparatus, comprising:
processing circuitry configured to
obtain one or more second images based on a first image, each of the one or more second images having a first size;
perform frequency domain transformation on the one or more second images to obtain one or more first spectrograms of the one or more second images;
perform down-sampling processing on each of the one or more first spectrograms to obtain one or more second spectrograms of the one or more second images, the one or more second spectrograms corresponding to a spectral feature of the first image, the spectral feature representing a distribution of a grayscale gradient in the first image, and each of the one or more second spectrograms having a second size less than the first size;
obtain one or more blur kernels of the one or more second images based on application of a kernel prediction model to each of the one or more second spectrograms of the one or more second images, each of the one or more blur kernels being a convolution kernel;
generate one or more sub-super-resolved images based on the one or more second images and the one or more blur kernels; and
generate a super-resolved image based on the one or more sub-super-resolved images, a definition of the super-resolved image being higher than a definition of the first image.
15 . The apparatus according to claim 14 , wherein the processing circuitry is configured to:
based on a determination that an image size of the first image is greater than the first size,
perform image cropping on the first image based on the first size to obtain at least two second images;
perform pixel averaging processing on an intersection area of at least two sub-super-resolved images corresponding to the at least two second images; and
fuse the at least two sub-super-resolved images based on the processed intersection area to generate the super-resolved image.
16 . The apparatus according to claim 14 , wherein the processing circuitry is configured to:
based on a determination that an image size of the first image is less than the first size,
perform image expansion on the first image based on the first size to obtain a single second image, the image expansion including copying and stitching processing of the first image; and
perform image cropping on a single sub-super-resolved image corresponding to the single second image to generate the super-resolved image.
17 . The apparatus according to claim 14 , wherein the processing circuitry is configured to:
perform discrete fast Fourier transform on the one or more second images to obtain the one or more first spectrograms.
18 . The apparatus according to claim 14 , wherein
the kernel prediction model includes at least one convolution layer and at least one transposed convolution layer, a number of the at least one convolution layer being consistent with a number of the at least one transposed convolution layer; and
the processing circuitry is configured to:
perform convolution processing on the corresponding second spectrogram through the at least one convolution layer in the kernel prediction model to obtain a feature vector of the corresponding second spectrogram; and
perform transposed convolution processing on the feature vector through the at least one transposed convolution layer to output the corresponding blur kernel.
19 . The apparatus according to claim 14 , wherein
the one or more sub-super-resolved images are generated based on a non-blind super-resolution model according to a spatial feature transform for multiple degradations (SFTMD) network or a residual channel attention network (RCAN).