Image processing methods and apparatus, and electronic devices
View Patent ↗Image processing methods, apparatuses, and electronic devices include: extracting features of an image to be processed to obtain a first feature map of the image; generating an attention map of the image based on the first feature map; fusing the attention map and the first feature map to obtain a fusion map; and extracting the features of the image again based on the fusion map. The implementation mode introduces an attention mechanism into image processing, and effectively improves the efficiency of acquiring information from an image.
1. An image processing method, comprising:
extracting features of an image to be processed to obtain a first feature map of the image;
generating an attention map of the image based on the first feature map;
fusing the attention map and the first feature map to obtain a fusion map; and
extracting features of the image again based on the fusion map.
2. The method according to claim 1 , wherein the generating an attention map of the image based on the first feature map comprises:
sequentially performing downsampling processing on the first feature map N times, wherein N is an integer greater than or equal to 1; and
sequentially performing upsampling processing on a feature map subjected to a N th downsampling processing N times to obtain the attention map of the image, wherein a resolution of the attention map is the same as that of the first feature map.
3. The method according to claim 2 , wherein the sequentially performing upsampling processing on the feature map subjected to the N th downsampling processing N times comprises:
performing a convolution operation on a feature map subjected to a (N−n) th downsampling processing and a feature map subjected to a n th upsampling processing, wherein n is an integer greater than 1 and less than N; and
performing a (n+1) th upsampling processing on a feature map subjected to the convolution operation.
4. The method according to claim 3 , wherein the performing a convolution operation on a feature map subjected to a (N−n) th downsampling processing and a feature map subjected to the n th upsampling processing comprises:
performing convolution processing on the feature map subjected to the (N−n) th downsampling processing to obtain a convolution map;
adding a feature value of at least one pixel in the convolution map to a feature value of a corresponding pixel in the feature map subjected to the n th upsampling processing; and
performing the convolution operation on the feature map subjected to the addition.
5. The method according to claim 3 , wherein the sequentially performing upsampling processing on the feature map subjected to the N th downsampling processing N times further comprises:
performing at least one convolution operation on the feature map subjected to the N th downsampling processing; and
performing a first upsampling processing on a feature map subjected to a last convolution operation.
6. The method according to claim 1 , wherein the fusing the attention map and the first feature map to obtain a fusion map comprises:
performing at least one convolution operation on the first feature map; and
fusing the attention map and the first feature map subjected to the last convolution operation to obtain the fusion map.
7. The method according to claim 1 , wherein the fusing the attention map and the first feature map to obtain a fusion map comprises:
performing normalization processing on the attention map; and
fusing the attention map subjected to the normalization processing and the first feature map to obtain the fusion map.
8. The method according to claim 7 , wherein the performing normalization processing on the attention map comprises:
sequentially performing at least one convolution processing on the attention map; and
performing normalization processing on the attention map subjected to the last convolution processing.
9. The method according to claim 7 , wherein the fusing the attention map and the first feature map to obtain a fusion map comprises:
multiplying a weight value of at least one pixel in the attention map subjected to the normalization processing by a feature value of a corresponding pixel in the first feature map to obtain the fusion map.
10. The method according to claim 7 , wherein the fusing the attention map and the first feature map to obtain a fusion map comprises:
multiplying a weight value of at least one pixel in the attention map subjected to the normalization processing by a feature value of a corresponding pixel in the first feature map to obtain a multiplication map; and
adding the feature value of at least one pixel in the multiplication map to the feature value of the corresponding pixel in the first feature map to obtain the fusion map.
11. The method according to claim 1 , wherein after the extracting the features of the image again based on the fusion map, the method further comprises at least one of the following:
detecting or identifying an object included in the image according to the features of the image extracted again;
determining the category of the object included in the image according to the features of the image extracted again; and
segmenting the image according to the features of the image extracted again.
12. An electronic apparatus, comprising:
a processor; and
a memory for storing instructions executable by the processor;
wherein the processor executes the instructions to implement the following operations:
extracting features of an image to be processed to obtain a first feature map of the image;
generating an attention map of the image based on the first feature map;
fusing the attention map and the first feature map to obtain a fusion map; and
extracting features of the image again based on the fusion map.
13. The apparatus according to claim 12 , wherein the operation of generating an attention map of the image based on the first feature map comprises:
sequentially performing downsampling processing on the first feature map N times, wherein N is an integer greater than or equal to 1; and
sequentially performing upsampling processing on a feature map subjected to a N th downsampling processing N times to obtain the attention map of the image, wherein a resolution of the attention map is the same as that of the first feature map.
14. The apparatus according to claim 13 , wherein the operation of sequentially performing upsampling processing on the feature map subjected to the N th downsampling processing N times to obtain the attention map of the image comprises:
performing a convolution operation on a feature map subjected to the (N−n) th downsampling processing and a feature map subjected to a n th upsampling processing, wherein n is an integer greater than 1 and less than N; and
performing a (n+1) th upsampling processing on a feature map subjected to the convolution operation to obtain the attention map of the image.
15. The apparatus according to claim 14 , wherein the operation of performing a convolution operation on a feature map subjected to a (N−n) th downsampling processing and a feature map subjected to the n th upsampling processing comprises:
performing convolution processing on the feature map subjected to the (N−n) th downsampling processing to obtain a convolution map;
adding the feature value of at least one pixel in the convolution map to the feature value of a corresponding pixel in the feature map subjected to the n th upsampling processing; and
performing the convolution operation on the feature map subjected to the addition.
16. The apparatus according to claim 14 , wherein the operation of sequentially performing upsampling processing on the feature map subjected to the N th downsampling processing N times further comprises:
performing at least one convolution operation on the feature map subjected to the N th downsampling processing; and
performing a first upsampling processing on a feature map subjected to a last convolution operation to obtain the attention map of the image.
17. The apparatus according to claim 14 , wherein the processor executes the instructions to implement at least one of the following:
detecting or identifying an object included in the image according to the features of the image extracted again;
determining the category of the object included in the image according to the features of the image extracted again; and
segmenting the image according to the features of the image extracted again.
18. The apparatus according to claim 12 , wherein the operation of fusing the attention map and the first feature map to obtain a fusion map comprises:
performing at least one convolution operation on the first feature map; and
fusing the attention map and the first feature map subjected to the last convolution operation to obtain the fusion map;
or,
performing normalization processing on the attention map; and
fusing the attention map subjected to the normalization processing and the first feature map to obtain the fusion map.
19. The apparatus according to claim 18 , wherein the operation of performing normalization processing on the attention map comprises:
sequentially performing at least one convolution processing on the attention map; and
performing normalization processing on the attention map subjected to the last convolution processing.
20. A non-transitory computer-readable storage medium, having computer instructions stored thereon, wherein execution of the computer-readable instructions by a processor causes the processor to implement:
extracting features of an image to be processed to obtain a first feature map of the image;
generating an attention map of the image based on the first feature map;
fusing the attention map and the first feature map to obtain a fusion map; and
extracting features of the image again based on the fusion map.