Visual analysis methods and apparatuses for image inpainting models, and electronic devices
The present disclosure provides a visual analysis method and apparatus for an image inpainting model, and an electronic device. The method includes: obtaining a benchmark image and a plurality of interpolation images corresponding to a to-be-restored image, where the to-be-restored image includes a target region to be restored; inputting the to-be-restored image, the benchmark image and the plurality of the interpolation images as input images into the image inpainting model, to obtain a restored image corresponding to each of the input images; obtaining an attribution value of the target region in the restored image corresponding to each of the input images; based on the attribution value, obtaining an activation image corresponding to each of the input images, where the activation image includes a response value of each pixel in an input image contributing to the attribution value.
1 . A method, comprising:
obtaining a benchmark image corresponding to a to-be-restored image and a plurality of interpolation images corresponding to the to-be-restored image, wherein the to-be-restored image comprises a target region to be restored;
respectively inputting the to-be-restored image, the benchmark image and the plurality of the interpolation images as input images into an image inpainting model, to obtain a restored image corresponding to each of the input images;
for each of the input images,
obtaining an attribution value of the target region in the restored image corresponding to the input image, and
based on the attribution value, obtaining an activation image corresponding to the input image, wherein the activation image comprises a response value of each pixel in the input image contributing to the attribution value; and
performing visual analysis based on the activation image corresponding to each of the input images and the restored image corresponding to the to-be-restored image, to obtain a visual analysis result;
wherein obtaining the benchmark image corresponding to the to-be-restored image comprises:
filtering the to-be-restored image by a preset filtering model, to obtain blurred images corresponding to the to-be-restored image;
obtaining degrees of blurriness of the blurred images; and
determining a blurred image with a degree of blurriness exceeding a blurriness threshold as the benchmark image corresponding to the restored image;
wherein the degrees of blurriness of the blurred images can be calculated by:
g
¯
=
1
N
x
×
N
y
∑
x
=
1
N
x
∑
y
=
1
N
y
f
(
x
,
y
)
,
and
s
=
1
N
x
×
N
y
∑
x
=
1
N
x
∑
y
=
1
N
y
(
f
(
x
,
y
)
-
g
_
)
2
,
wherein x represents an abscissa, y represents an ordinate, f (x, y) represents a grayscale value of a pixel (x, y), Nx represents a row number of the pixel, Ny represents a column number of the pixel, g represents a grayscale average value, and s represents a grayscale variance;
wherein obtaining the plurality of the interpolation images corresponding to the to-be-restored image comprises:
based on the to-be-restored image and an preset path function, obtaining the plurality of the interpolation images corresponding to the to-be-restored image, wherein the plurality of the interpolation images have different degrees of blurriness; wherein the preset path function satisfies that a starting image of a path is the benchmark image, and an ending image of the path is the to-be-restored image;
wherein the preset path function is represented by:
γ(α)=ω(σ−ασ)⊗I;
wherein I represents the to-be-restored image, ⊗ represents a convolution operation, a value range of α is 0<α<1, ω(σ) represents a Gaussian kernel with a convolution kernel size of σ, and γ(α) represents an interpolation image at α on the path, wherein the preset path function satisfies γ pb (0)=I′ and γ pb (1)=I, and I′ represents the blurred image;
wherein obtaining the attribution value comprises:
by a gradient detection model, a detection value of the target region in the restored image corresponding to the input image as the attribution value; wherein the gradient detection model is configured to detect high-frequency information contained in an image region;
wherein the gradient detection model is F l (x, y): , where l represents a patch size of a restored image corresponding to each input image, F l (x, y) represents a number of high-frequency components contained in a target region with (x, y) being a center and width and height being I, wherein the larger the value of F l (x, y), the more high-frequency components in the target region;
wherein performing the visual analysis comprises:
integrating the activation image corresponding to each of the input images to obtain a target activation image corresponding to the to-be-restored image, wherein the target activation image represents a region of pixels of interest when the image inpainting model restores the to-be-restored image; and
based on the target activation image, performing visual analysis on the restored image corresponding to the to-be-restored image, to obtain the visual analysis result;
wherein the integration is performed by:
Path
I
G
i
(
F
,
u
,
γ
)
=
∫
α
=
0
1
∂
F
(
u
(
γ
(
α
)
)
)
∂
γ
(
α
)
i
×
∂
γ
(
α
)
i
∂
α
d
α
;
wherein PathIG represents a path gradient integral value, PathIG i represents a contribution value of pixel i, u represents an image inpainting model, γ(α) represents an interpolation image at α on the path, F represents an attribution function wherein a response value of each pixel on an input image is obtained based on the attribution value, u(γ(α)) represents the restored image corresponding to the interpolation image.
2 . The method according to claim 1 , wherein a response value of a pixel of interest in the target activation image is directly proportional to a degree of attention given to a position corresponding to the pixel of interest when the image inpainting model restores the to-be-restored image.
3 . A electronic device, comprising:
one or more memories and one or more processors, wherein the one or more memories are configured to store a computer program that is executable by the one or more processors, and
the one or more processors are configured to execute the computer program in the one or more memories, to implement a method, the method comprising:
obtaining a benchmark image corresponding to a to-be-restored image and a plurality of interpolation images corresponding to the to-be-restored image, wherein the to-be-restored image comprises a target region to be restored;
respectively inputting the to-be-restored image, the benchmark image and the plurality of the interpolation images as input images into an image inpainting model, to obtain a restored image corresponding to each of the input images;
for each of the input images,
obtaining an attribution value of the target region in the restored image corresponding to the input image, and
based on the attribution value, obtaining an activation image corresponding to the input image, wherein the activation image comprises a response value of each pixel in the input image contributing to the attribution value; and
performing visual analysis based on the activation image corresponding to each of the input images and the restored image corresponding to the to-be-restored image, to obtain a visual analysis result;
wherein obtaining the benchmark image corresponding to the to-be-restored image comprises:
filtering the to-be-restored image by a preset filtering model, to obtain blurred images corresponding to the to-be-restored image;
obtaining degrees of blurriness of the blurred images; and
determining a blurred image with a degree of blurriness exceeding a blurriness threshold as the benchmark image corresponding to the restored image;
wherein the degrees of blurriness of the blurred images can be calculated by:
g
¯
=
1
N
x
×
N
y
∑
x
=
1
N
x
∑
y
=
1
N
y
f
(
x
,
y
)
,
and
s
=
1
N
x
×
N
y
∑
x
=
1
N
x
∑
y
=
1
N
y
(
f
(
x
,
y
)
-
g
_
)
2
,
wherein x represents an abscissa, y represents an ordinate, f (x, y) represents a grayscale value of a pixel (x, y), Nx represents a row number of the pixel, Ny represents a column number of the pixel, g represents a grayscale average value, and s represents a grayscale variance;
wherein obtaining the plurality of the interpolation images corresponding to the to-be-restored image comprises:
based on the to-be-restored image and an preset path function, obtaining the plurality of the interpolation images corresponding to the to-be-restored image, wherein the plurality of the interpolation images have different degrees of blurriness; wherein the preset path function satisfies that a starting image of a path is the benchmark image, and an ending image of the path is the to-be-restored image;
wherein the preset path function is represented by:
γ(α)=ω(σ−ασ)⊗I;
wherein I represents the to-be-restored image, ⊗ represents a convolution operation, a value range of α is 0<α<1, ω(σ) represents a Gaussian kernel with a convolution kernel size of σ, and γ(α) represents an interpolation image at α on the path, wherein the preset path function satisfies γ pb (0)=I′ and γ pb (1)=I, and I′ represents the blurred image;
wherein obtaining the attribution value comprises:
by a gradient detection model, a detection value of the target region in the restored image corresponding to the input image as the attribution value; wherein the gradient detection model is configured to detect high-frequency information contained in an image region;
wherein the gradient detection model is F l (x, y): , where l represents a patch size of a restored image corresponding to each input image, F l (x, y) represents a number of high-frequency components contained in a target region with (x, y) being a center and width and height being I, wherein the larger the value of F l (x, y), the more high-frequency components in the target region;
wherein performing the visual analysis comprises:
integrating the activation image corresponding to each of the input images to obtain a target activation image corresponding to the to-be-restored image, wherein the target activation image represents a region of pixels of interest when the image inpainting model restores the to-be-restored image; and
based on the target activation image, performing visual analysis on the restored image corresponding to the to-be-restored image, to obtain the visual analysis result;
wherein the integration is performed by:
Path
I
G
i
(
F
,
u
,
γ
)
=
∫
α
=
0
1
∂
F
(
u
(
γ
(
α
)
)
)
∂
γ
(
α
)
i
×
∂
γ
(
α
)
i
∂
α
d
α
;
wherein PathIG represents a path gradient integral value, PathIG i represents a contribution value of pixel i, u represents an image inpainting model, γ(α) represents an interpolation image at α on the path, F represents an attribution function wherein a response value of each pixel on an input image is obtained based on the attribution value, u(γ(α)) represents the restored image corresponding to the interpolation image.
4 . A non-transitory computer-readable storage medium, with an executable computer program stored therein, wherein when the executable computer program in the storage medium is executed by a processor, a method is implemented, wherein the method comprising:
obtaining a benchmark image corresponding to a to-be-restored image and a plurality of interpolation images corresponding to the to-be-restored image, wherein the to-be-restored image comprises a target region to be restored;
respectively inputting the to-be-restored image, the benchmark image and the plurality of the interpolation images as input images into an image inpainting model, to obtain a restored image corresponding to each of the input images;
for each of the input images,
obtaining an attribution value of the target region in the restored image corresponding to the input image, and
based on the attribution value, obtaining an activation image corresponding to the input image, wherein the activation image comprises a response value of each pixel in the input image contributing to the attribution value; and
performing visual analysis based on the activation image corresponding to each of the input images and the restored image corresponding to the to-be-restored image, to obtain a visual analysis result;
wherein obtaining the benchmark image corresponding to the to-be-restored image comprises:
filtering the to-be-restored image by a preset filtering model, to obtain blurred images corresponding to the to-be-restored image;
obtaining degrees of blurriness of the blurred images; and
determining a blurred image with a degree of blurriness exceeding a blurriness threshold as the benchmark image corresponding to the restored image;
wherein the degrees of blurriness of the blurred images can be calculated by:
g
¯
=
1
N
x
×
N
y
∑
x
=
1
N
x
∑
y
=
1
N
y
f
(
x
,
y
)
,
and
s
=
1
N
x
×
N
y
∑
x
=
1
N
x
∑
y
=
1
N
y
(
f
(
x
,
y
)
-
g
_
)
2
,
wherein x represents an abscissa, y represents an ordinate, f (x, y) represents a grayscale value of a pixel (x, y), Nx represents a row number of the pixel, Ny represents a column number of the pixel, g represents a grayscale average value, and s represents a grayscale variance;
wherein obtaining the plurality of the interpolation images corresponding to the to-be-restored image comprises:
based on the to-be-restored image and an preset path function, obtaining the plurality of the interpolation images corresponding to the to-be-restored image, wherein the plurality of the interpolation images have different degrees of blurriness; wherein the preset path function satisfies that a starting image of a path is the benchmark image, and an ending image of the path is the to-be-restored image;
wherein the preset path function is represented by:
γ(α)=ω(σ−ασ)⊗I;
wherein I represents the to-be-restored image, ⊗ represents a convolution operation, a value range of α is 0<α<1, ω(σ) represents a Gaussian kernel with a convolution kernel size of σ, and γ(α) represents an interpolation image at a on the path, wherein the preset path function satisfies γ pb (0)=I′ and γ pb (1)=I, and I′ represents the blurred image;
wherein obtaining the attribution value comprises:
by a gradient detection model, a detection value of the target region in the restored image corresponding to the input image as the attribution value; wherein the gradient detection model is configured to detect high-frequency information contained in an image region;
wherein the gradient detection model is F l (x, y): , where l represents a patch size of a restored image corresponding to each input image, F l (x, y) represents a number of high-frequency components contained in a target region with (x, y) being a center and width and height being I, wherein the larger the value of F l (x, y), the more high-frequency components in the target region;
wherein performing the visual analysis comprises:
integrating the activation image corresponding to each of the input images to obtain a target activation image corresponding to the to-be-restored image, wherein the target activation image represents a region of pixels of interest when the image inpainting model restores the to-be-restored image; and
based on the target activation image, performing visual analysis on the restored image corresponding to the to-be-restored image, to obtain the visual analysis result;
wherein the integration is performed by:
Path
I
G
i
(
F
,
u
,
γ
)
=
∫
α
=
0
1
∂
F
(
u
(
γ
(
α
)
)
)
∂
γ
(
α
)
i
×
∂
γ
(
α
)
i
∂
α
d
α
;
wherein PathIG represents a path gradient integral value, PathIG i represents a contribution value of pixel i, u represents an image inpainting model, γ(α) represents an interpolation image at α on the path, F represents an attribution function wherein a response value of each pixel on an input image is obtained based on the attribution value, u(γ(α)) represents the restored image corresponding to the interpolation image.
5 . The electronic device according to claim 4 , wherein a response value of a pixel of interest in the target activation image is directly proportional to a degree of attention given to a position corresponding to the pixel of interest when the image inpainting model restores the to-be-restored image.
6 . The storage medium according to claim 4 , wherein obtaining the benchmark image corresponding to the to-be-restored image comprises:
filtering the to-be-restored image by a preset filtering model, to obtain blurred images corresponding to the to-be-restored image;
obtaining degrees of blurriness of the blurred images; and
determining a blurred image with a degree of blurriness exceeding a blurriness threshold as the benchmark image corresponding to the restored image.
7 . The storage medium according to claim 6 , wherein obtaining the plurality of the interpolation images corresponding to the to-be-restored image comprises:
based on the to-be-restored image and a preset path function, obtaining the plurality of the interpolation images corresponding to the to-be-restored image, wherein the plurality of the interpolation images have different degrees of blurriness; wherein the preset path function satisfies that a starting image of a path is the benchmark image, and an ending image of the path is the to-be-restored image.
8 . The storage medium according to claim 4 , wherein obtaining the attribution value comprises:
by a gradient detection model, a detection value of the target region in the restored image corresponding to the input image as the attribution value; wherein the gradient detection model is configured to detect high-frequency information contained in an image region.
9 . The storage medium according to claim 4 , wherein performing the visual analysis comprises:
integrating the activation image corresponding to each of the input images to obtain a target activation image corresponding to the to-be-restored image, wherein the target activation image represents a region of pixels of interest when the image inpainting model restores the to-be-restored image; and
based on the target activation image, performing visual analysis on the restored image corresponding to the to-be-restored image, to obtain the visual analysis result.
10 . The storage medium according to claim 9 , wherein a response value of a pixel of interest in the target activation image is directly proportional to a degree of attention given to a position corresponding to the pixel of interest when the image inpainting model restores the to-be-restored image.
11 . The method according to claim 1 , wherein the visual analysis result is a region of pixels of interest when the image inpainting model restores the to-be-restored image, where the region of pixels of interest is determined by analysing a contribution of each pixel to a restoration result of the to-be-restored image.
12 . The electronic device according to claim 3 , wherein the visual analysis result is a region of pixels of interest when the image inpainting model restores the to-be-restored image, where the region of pixels of interest is determined by analysing a contribution of each pixel to a restoration result of the to-be-restored image.