Electronic device for estimating optical flow and operating method thereof

1 . A method of estimating an optical flow, the method comprising:

processing, using an image processing pass, a first image and a second image separately; and

estimating the optical flow based on a second image attention feature map of the first image processed using the image processing pass and a second image attention feature map of the second image processed using the image processing pass,

wherein the processing of an image using the image processing pass comprises:

extracting an image feature map of the image by encoding the image;

outputting a first image fusion attention feature map by fusing image width-axial row relationship information of the image with the image feature map;

outputting a first image attention feature map of the image based on the first image fusion attention feature map and the image feature map;

outputting a second image fusion attention feature map by fusing image height-axial column relationship information of the image with the first image attention feature map including the image width-axial row relationship information; and

generating a second image attention feature map of the image by adding the second image fusion attention feature map and the first image attention feature map.

2 . The method of claim 1 , wherein the estimating of the optical flow comprises:

generating a third image fusion attention feature map of the first image by fusing second row relationship information of the second image with the second image attention feature map of the first image based on the second image attention feature map of the first image and an image feature map of the second image;

obtaining a third image attention feature map of the first image based on the third image fusion attention feature map of the first image and the second image attention feature map of the first image;

generating a fourth image fusion attention feature map of the first image by fusing second column relationship information of the second image with the third image attention feature map; and

obtaining a fourth image attention feature map of the first image based on the fourth image fusion attention feature map of the first image and the third image attention feature map of the first image.

3 . The method of claim 2 , wherein the estimating of the optical flow further comprises:

generating a third image fusion attention feature map of the second image by fusing first row relationship information of the first image with the second image attention feature map of the second image based on the second image attention feature map of the second image and an image feature map of the first image;

obtaining a third image attention feature map of the second image based on the third image fusion attention feature map of the second image and the second image attention feature map of the second image;

generating a fourth image fusion attention feature map of the second image by fusing first column relationship information of the first image with the third image attention feature map; and

obtaining a fourth image attention feature map of the second image based on the fourth image fusion attention feature map of the second image and the third image attention feature map of the second image.

4 . The method of claim 3 , wherein the estimating of the optical flow further comprises estimating the optical flow based on the fourth image attention feature map of the first image and the fourth image attention feature map of the second image.

5 . The method of claim 1 , wherein the processing of the image using the image processing pass further comprises:

concatenating a row vector for extracting the image width-axial row relationship information with the image feature map of the image;

obtaining a first vector feature map of the image based on the image feature map of the image concatenated with the row vector, wherein the first vector feature map is width-axially separated and attention-learnable;

separating the first vector feature map into a width-axially separated feature map of the image and a first information map corresponding to the image width axial row relationship information; and

generating the first image fusion attention feature map based on the width-axially separated feature map of the image and the first information map.

6 . The method of claim 1 , wherein the processing of the image using the image processing pass further comprises:

concatenating a column vector for extracting the image height axial column relationship information with the first image attention feature map of the image;

obtaining a second vector feature map of the image based on the first image attention feature map of the image concatenated with the column vector, wherein the second vector feature map is height-axially separated and attention-learnable;

separating the second vector feature map into a height-axially separated feature map of the image and a second information map corresponding to the image height axial column relationship information; and

generating the second image fusion attention feature map based on the height-axially separated feature map of the image and the second information map.

7 . An apparatus for estimating an optical flow, comprising:

a memory configured to store one or more instructions; and

a processor communicatively coupled to the memory,

wherein the processor is configured to execute the one or more instructions to:

process, using an image processing pass, a first image and a second image separately; and

estimate the optical flow based on a second image attention feature map of the first image obtained using the image processing pass and a second image attention feature map of the second image obtained using the image processing pass,

wherein the image processing pass is configured to:

extract an image feature map of an image by encoding the image;

output a first image fusion attention feature map by fusing image width-axial row relationship information of the image with the image feature map;

output a first image attention feature map of the image based on the first image fusion attention feature map and the image feature map;

output a second image fusion attention feature map by fusing image height-axial column relationship information of the image with the first image attention feature map including the image width-axial row relationship information; and

generate a second image attention feature map of the image by adding the second image fusion attention feature map and the first image attention feature map.

8 . The apparatus of claim 7 , wherein the processor is further configured to execute the one or more instructions to:

generate a third image fusion attention feature map of the first image by fusing second row relationship information of the second image with the second image attention feature map of the first image based on the second image attention feature map of the first image and an image feature map of the second image;

obtain a third image attention feature map of the first image based on the third image fusion attention feature map of the first image and the second image attention feature map of the first image;

generate a fourth image fusion attention feature map of the first image by fusing second column relationship information of the second image with the third image attention feature map; and

obtain a fourth image attention feature map of the first image based on the fourth image fusion attention feature map of the first image and the third image attention feature map of the first image.

9 . The apparatus of claim 8 , wherein the processor is further configured to execute the one or more instructions to:

generate a third image fusion attention feature map of the second image by fusing first row relationship information of the first image with the second image attention feature map of the second image based on the second image attention feature map of the second image and an image feature map of the first image;

obtain a third image attention feature map of the second image based on the third image fusion attention feature map of the second image and the second image attention feature map of the second image;

generate a fourth image fusion attention feature map of the second image by fusing first column relationship information of the first image with the third image attention feature map; and

obtain a fourth image attention feature map of the second image based on the fourth image fusion attention feature map of the second image and the third image attention feature map of the second image.

10 . The apparatus of claim 9 , wherein the processor is further configured to execute the one or more instructions to estimate the optical flow based on the fourth image attention feature map of the first image and the fourth image attention feature map of the second image.

11 . The apparatus of claim 7 , wherein the image processing pass is further configured to:

concatenate a row vector for extracting the image width-axial row relationship information with the image feature map of the image;

obtain a first vector feature map of the image based on the image feature map of the image concatenated with the row vector, wherein the first vector feature map is width-axially separated and attention-learnable;

separate the first vector feature map into a width-axially separated feature map of the image and a first information map corresponding to the image width axial row relationship information; and

generate the first image fusion attention feature map based on the width-axially separated feature map of the image and the first information map.

12 . The apparatus of claim 7 , wherein the image processing pass is further configured to:

concatenate a column vector for extracting the image height axial column relationship information with the first image attention feature map of the image;

obtain a second vector feature map of the image based on the first image attention feature map of the image concatenated with the column vector, wherein the second vector feature map is height-axially separated and attention-learnable;

separate the second vector feature map into a height-axially separated feature map of the image and a second information map corresponding to the image height axial column relationship information; and

generate the second image fusion attention feature map based on the height-axially separated feature map of the image and the second information map.

13 . A method of estimating an optical flow, the method comprising:

generating a plurality of first clusters comprising features of a first image feature map representing a first image by clustering the first image feature map;

determining a pixel in a second image corresponding to a center feature from among the features respectively included in the plurality of first clusters from a second image feature map representing the second image, wherein the second image feature map is obtained by

outputting a first image fusion attention feature map by fusing an image width-axial row relationship information of the first image with the first image feature map;

outputting a first image attention feature map of the first image based on the first image fusion attention feature map and the first image feature map;

outputting a second image fusion attention feature map by fusing image height-axial column relationship information of the first image with the first image attention feature map including the image width-axial row relationship information; and

generating, as the second image feature map, a second image attention feature map of the first image by adding the second image fusion attention feature map and the first image attention feature map;

obtaining a loss matrix based on the plurality of first clusters and the determined pixel in the second image; and

estimating the optical flow of the first image and the second image based on the loss matrix.

14 . The method of claim 13 , wherein the generating of the plurality of first clusters comprises:

dividing the features of the first image feature map of the first image into a plurality of subsets; and

dividing the features of the first image feature map of the first image into the plurality of first clusters.

15 . The method of claim 13 , wherein the determining of the pixel in the second image comprises:

determining one or more features most associated with the center feature of each cluster among the plurality of first clusters in an image feature matrix of the second image.

16 . The method of claim 13 , wherein the obtaining of the loss matrix comprises calculating a correlation between the center feature and the pixel in the second image.

17 . An apparatus for estimating an optical flow, comprising:

a memory configured to store one or more instructions; and

a processor communicatively coupled to the memory,

wherein the processor is configured to execute the one or more instructions to:

generate a plurality of first clusters comprising features of a first feature map representing a first image by clustering the first feature map;

determine a pixel in a second image corresponding to a center feature from among the features respectively included in the plurality of first clusters from a second feature map representing the second image, wherein the second image feature map is obtained by

outputting a first image fusion attention feature map by fusing an image width-axial row relationship information of the first image with the first image feature map;

outputting a first image attention feature map of the first image by using the first image fusion attention feature map and the first image feature map as inputs for the second process;

outputting a second image fusion attention feature map by fusing image height-axial column relationship information of the first image with the first image attention feature map including the image width-axial row relationship information; and

generating, as the second image feature map, a second image attention feature map of the first image by adding the second image fusion attention feature map and the first image attention feature map as inputs for the fourth process;

obtain a loss matrix based on the plurality of first clusters and the determined pixel in the second image; and

estimate the optical flow of the first image and the second image based on the loss matrix.

18 . The apparatus of claim 17 , wherein the processor is further configured to execute the one or more instructions to:

divide the features of the first feature map of the first image into a plurality of subsets; and

divide the features of the first feature map of the first image into the plurality of first clusters.

19 . The apparatus of claim 17 , wherein the processor is further configured to execute the one or more instructions to:

determine one or more features most associated with the center feature of each cluster among the plurality of first clusters in an image feature matrix of the second image.

20 . The apparatus of claim 17 , wherein the processor is further configured to execute the one or more instructions to:

calculate a correlation between the center feature and the pixel in the second image.