Multi-scale detail enhancement model for images
An image processing method, comprising: by using a trained multi-scale detail enhancement model, performing detail enhancement on an input image to be processed; wherein multi-scale decomposition is performed on the input image to obtain a base layer image and at least one detail layer image; a first residual feature corresponding to a first feature map is acquired, and a second residual feature corresponding to a second feature map of each detail layer image is acquired; a base layer output image is obtained according to the first residual feature, each second residual feature and the first feature map, and a detail layer output image corresponding to the detail layer image is obtained according to the first residual feature, each second residual feature and the second feature map; and image fusion is performed on the base layer output image and each detail layer output image to obtain an output image.
1 . An image processing method, comprising:
performing detail enhancement on an input image to be processed to obtain an output image by using a trained multi-scale detail enhancement model;
wherein the multi-scale detail enhancement model is configured to perform following acts:
performing a multi-scale decomposition on the input image to obtain a base layer image and at least one detail layer image of the input image;
acquiring a first feature map of the base layer image and acquiring a second feature map of each of the at least one detail layer image;
acquiring a first residual feature corresponding to the first feature map, and acquiring a second residual feature corresponding to the second feature map of each of the at least one detail layer image; obtaining a base layer output image according to the first residual feature, each of at least one second residual feature, and the first feature map, and
for each of the at least one detail layer image, obtaining a detail layer output image corresponding to each of the at least one detail layer image according to the first residual feature, each of the at least one second residual feature and the second feature map of the each of the at least one detail layer image; and
performing an image fusion on the base layer output image and each of detail layer output images to obtain the output image,
wherein the acquiring the first residual feature corresponding to the first feature map and acquiring the second residual feature corresponding to the second feature map of each of the at least one detail layer image comprises:
acquiring the first residual feature and each of the at least one second residual feature through P stage operation blocks which are connected sequentially, wherein P is a positive integer;
wherein, for a first stage operation block, taking the first feature map and each of at least one second feature map as inputs, extracting a residual feature of the first feature map and a residual feature of each of the at least one second feature map, superimposing the residual feature of the first feature map and the residual feature of each of the at least one second feature map to obtain a multi-scale residual feature corresponding to the first stage operation block;
for a m-th stage operation block, wherein m is a positive integer greater than 1 and m is not greater than P, taking a first intermediate feature map and each of at least one second intermediate feature map outputted by a (m−1)-th stage operation block as inputs, extracting a residual feature of the first intermediate feature map and a residual feature of each of the at least one second intermediate feature map, superimposing the residual feature of the first intermediate feature map and the residual feature of each of the at least one second intermediate feature map to obtain a multi-scale residual feature corresponding to the m-th stage operation block;
the obtaining a base layer output image according to the first residual feature, each of the at least one second residual feature and the first feature map, and for each of the at least one detail layer image, obtaining a detail layer output image corresponding to each of the at least one detail layer image according to the first residual feature, each of the at least one second residual feature and the second feature map of the each of the at least one detail layer image comprises:
obtaining the base layer output image according to the multi-scale residual feature obtained by each of the P stage operation blocks which are connected sequentially and the first feature map, and obtaining the detail layer output image corresponding to each of the at least one detail layer image according to the multi-scale residual feature obtained by each of the P stage operation blocks which are connected sequentially and the second feature map of each of the at least one detail layer image;
wherein, for the first stage operation block, superimposing the multi-scale residual feature corresponding to the first stage operation block and the first feature map to obtain and output the first intermediate feature map of the first stage operation block, and superimposing the multi-scale residual feature corresponding to the first stage operation block and each of the at least one second feature map, respectively, to obtain and output each of the at least one second intermediate feature map of the first stage operation block;
for the m-th stage operation block, superimposing the multi-scale residual feature corresponding to the m-th stage operation block and the first intermediate feature map outputted by the (m−1)-th stage operation block to obtain and output the first intermediate feature map of the m-th stage operation block, and superimposing the multi-scale residual feature corresponding to the m-th stage operation block and each of the at least one second intermediate feature map outputted by the (m−1)-th stage operation block, respectively, to obtain and output each of the at least one second intermediate feature map of the m-th stage operation block; and
obtaining the base layer output image according to the first intermediate feature map outputted by the P-th stage operation block, and obtaining the detail layer output image corresponding to each of the at least one detail layer image according to each of the at least one second intermediate feature map outputted by the P-th stage operation block, respectively.
2 . The image processing method according to claim 1 , wherein each stage operation block comprises a first operation unit and at least one second operation unit, and the first operation unit and the second operation unit each comprises at least two convolution layers;
wherein, for the first stage operation block, a residual feature of the first feature map is extracted by using the first operation unit of the first stage operation block, and a residual feature of each of the second feature maps is extracted by using the second operation unit of the first stage operation block; and
for the m-th stage operation block, a residual feature of the first intermediate feature map outputted by the (m−1)-th stage operation block is extracted by using the first operation unit of the m-th stage operation block, and a residual feature of each of the second intermediate feature maps outputted by the (m−1)th stage operation block is extracted by using the second operation unit of the m-th stage operation block.
3 . The image processing method according to claim 2 , wherein both the first operation unit and the second operation unit comprise a first convolution layer, a normalization layer, an activation layer, and a second convolution layer which are connected sequentially.
4 . The image processing method according to claim 3 , wherein the normalization layer is a half instance normalization layer.
5 . The image processing method according to claim 1 , wherein the multi-scale detail enhancement model is trained by following acts:
acquiring a first image sample and a second image sample corresponding to the first image sample, and performing the detail enhancement on the second image sample;
inputting the first image sample and the enhanced second image sample into the multi-scale detail enhancement model to be trained;
training the multi-scale detail enhancement model in an iteration manner based on the first image sample and the enhanced second image sample; and
ending the training to obtain the multi-scale detail enhancement model in response to a preset convergence condition being satisfied.
6 . The image processing method according to claim 5 , wherein the preset convergence condition comprises at least one of following:
having trained for a preset quantity of iterations; or
a loss value satisfying a preset loss value condition, wherein the loss value is obtained by a calculation based on the enhanced second image sample and the first image sample processed by the multi-scale detail enhancement model.
7 . The image processing method according to claim 5 , wherein the first image sample and the second image sample are noise-free images and noise images corresponding to a same image sample, respectively.
8 . The image processing method according to claim 1 , wherein the performing the multi-scale decomposition on the input image to obtain the base layer image and at least one detail layer image of the input image comprises:
performing an iterative filtering processing for n times on the input image, wherein n is a positive integer;
wherein a n-th filtering result is taken as the base layer image; and
for an i-th filtering result, wherein i is a positive integer and i is not greater than n, taking a difference between the i-th filtering result and the input image as the detail layer image, or taking a difference between the i-th filtering result and a (i-1)-th filtering result as the detail layer image.
9 . The image processing method according to claim 8 , wherein the performing an iterative filtering processing for n times on the input image comprises:
performing an iterative filtering processing for n times on the input image by using a edge preserving filtering operator.
10 . The image processing method according to claim 8 , wherein n is greater than or equal to 2.
11 . An electronic device, comprising:
one or more processors;
a memory for storing one or more programs; and
when the one or more programs are executed by the one or more processors, the one or more processors implementing the image processing method according to claim 1 .
12 . A non-transitory computer-readable medium, having a computer program stored thereon, wherein when the computer program is executed, the image processing method according to claim 1 is implemented.