Method and apparatus with image restoration

1 . A processor-implemented image restoration method, comprising:

receiving a burst image set comprising image frames;

determining shifted data representing a shift level of each of the image frames based on optical flow map sets generated using a shift estimation model;

selecting a base image from the image frames based on the shifted data; and

performing image restoration by synthesizing at least a portion of the image frames based on the base image.

2 . The image restoration method of claim 1 , wherein the determining of the shifted data comprises:

generating the optical flow map sets of the image frames comprising a first optical flow map set representing an optical flow between a first image frame of the image frames and another image frame of the image frames using the shift estimation model;

determining representative values of the optical flow map sets comprising a first representative value of the first optical flow map set; and

determining the shifted data based on the representative values.

3 . The image restoration method of claim 2 , wherein the first optical flow map set comprises:

optical flow maps comprising a first optical flow map representing a first optical flow between the first image frame and a second image frame of the image frames, and a second optical flow map representing a second optical flow between the first image frame and a third image frame of the image frames.

4 . The image restoration method of claim 3 , wherein the determining of the representative values comprises:

determining a first spatial mean value and a second spatial mean value according to spatial averaging of dimensions of the first optical flow map and the second optical flow map, respectively;

determining a pair-wise averaging result according to pair-wise averaging of the first spatial mean value and the second spatial mean value; and

determining a first representative value representing the first image frame from among the representative values based on the pair-wise averaging result.

5 . The image restoration method of claim 1 , further comprising:

determining noise data representing a noise level of each of the image frames using a noise estimation model.

6 . The image restoration method of claim 5 , wherein the selecting of the base image comprises:

selecting the base image from the image frames based on the shifted data and the noise data.

7 . The image restoration method of claim 5 , wherein each of the noise estimation model and the shift estimation model is based on a neural network.

8 . The image restoration method of claim 1 , further comprising:

determining a degradation score of each of the image frames based on the shifted data; and

forming a synthesis group with a portion of the image frames based on the degradation score of each of the image frames.

9 . The image restoration method of claim 8 , wherein the performing of the image restoration comprises:

synthesizing the image frames of the synthesis group based on the base image.

10 . The image restoration method of claim 1 , wherein the image frames comprise a first image frame and a second image frame, and the first image frame is selected as the base image, and

the performing of the image restoration comprises:

generating a first feature map of the first image frame;

generating an optical flow between the first image frame and the second image frame;

generating a second feature map of the second image frame;

generating a second warping result by warping the second feature map with the optical flow; and

fusing the first feature map and the second warping result.

11 . The image restoration method of claim 10 , wherein the performing of the image restoration further comprises:

determining a first weight of the first feature map and a second weight of the second feature map, and

the fusing of the first feature map and the second warping result comprises fusing the first feature map and the second warping result based on the first weight and the second weight.

12 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

13 . An image restoration apparatus, comprising:

a processor configured to:

receive a burst image set comprising image frames;

determine shifted data representing a shift level of each of the image frames based on optical flow map sets generated using a shift estimation model;

select a base image from the image frames based on the shifted data; and

perform image restoration by synthesizing at least a portion of the image frames based on the base image.

14 . The image restoration apparatus of claim 13 , wherein, to determine the shifted data, the processor is further configured to:

generate the optical flow map sets of the image frames comprising a first optical flow map set representing an optical flow between a first image frame of the image frames and another image frame of the image frames using the shift estimation model;

determine representative values of the optical flow map sets comprising a first representative value of the first optical flow map set; and

determine the shifted data based on the representative values.

15 . The image restoration apparatus of claim 14 , wherein the first optical flow map set comprises:

optical flow maps comprising a first optical flow map representing a first optical flow between the first image frame and a second image frame of the image frames, and a second optical flow map representing a second optical flow between the first image frame and a third image frame of the image frames.

16 . The image restoration apparatus of claim 15 , wherein, to determine the representative values, the processor is further configured to:

determine a first spatial mean value and a second spatial mean value according to spatial averaging of dimensions of the first optical flow map and the second optical flow map, respectively;

determine a pair-wise averaging result according to pair-wise averaging of the first spatial mean value and the second spatial mean value; and

determine a first representative value representing the first image frame from among the representative values based on the pair-wise averaging result.

17 . The image restoration apparatus of claim 13 , wherein the processor is further configured to:

determine noise data representing a noise level of each of the image frames using a noise estimation model; and

select the base image from the image frames based on the shifted data and the noise data.

18 . An electronic device, comprising:

a sensor configured to generate a burst image set comprising image frames; and

a processor configured to:

determine shifted data representing a shift level of each of the image frames based on optical flow map sets generated using a shift estimation model;

select a base image from the image frames based on the shifted data; and

perform image restoration by synthesizing at least a portion of the image frames based on the base image.

19 . The electronic device of claim 18 , wherein, to determine the shifted data, the processor is further configured to:

generate the optical flow map sets of the image frames comprising a first optical flow map set representing an optical flow between a first image frame of the image frames and another image frame of the image frames using the shift estimation model;

determine representative values of the optical flow map sets comprising a first representative value of the first optical flow map set; and

determine the shifted data based on the representative values.

20 . The electronic device of claim 19 , wherein the first optical flow map set comprises:

optical flow maps comprising a first optical flow map representing a first optical flow between the first image frame and a second image frame of the image frames, and a second optical flow map representing a second optical flow between the first image frame and a third image frame of the image frames, and

determine the representative values, the processor is further configured to:

determine a first spatial mean value and a second spatial mean value according to spatial averaging of dimensions of the first optical flow map and the second optical flow map, respectively;

determine a pair-wise averaging result according to pair-wise averaging of the first spatial mean value and the second spatial mean value; and

determine a first representative value representing the first image frame from among the representative values based on the pair-wise averaging result.