Burst image-based image restoration method and apparatus

1. An image restoration method comprising:

determining an anchor image based on individual images of a burst image set without aligning the individual images;

executing a feature extraction network based on the burst image set while using anchor information of the anchor image; and

generating a restored image based on a feature map corresponding to an output of the feature extraction network,

wherein the executing of the feature extraction network includes fusing the anchor information of the anchor image with features of the individual images,

wherein the executing of the feature extraction network comprises extracting a local feature while using the anchor information for each of a plurality of layer groups of the feature extraction network, and

wherein the executing of the local feature comprises:

extracting a primary local feature from a first individual image of the individual images using a first layer group of the plurality of layer groups;

transforming the primary local feature by fusing the anchor information with the primary local feature;

extracting a secondary local feature from the transformed primary local feature using a second layer group of the plurality of layer groups; and

determining a global feature based on the secondary local feature.

2. The method of claim 1 , wherein the determining of the anchor image comprises selecting one individual image, as the anchor image, among the individual images based on any one or any combination of any two or more of an image-deterioration-based selection, a time interval-based selection, and a random selection, and

wherein the image-deterioration-based selection is performed based on at least one quality feature of a minimum amount of noise/blur, a maximum signal-to-noise ratio (SNR), and a highest sharpness among the individual images.

3. The method of claim 1 , wherein the determining of the anchor image comprises generating a new individual image, as the anchor image, by applying weights to the individual images.

4. The method of claim 1 , wherein the anchor information comprises either one or both of image information of the anchor image and feature information of the anchor image.

5. The method of claim 1 , further comprising:

generating input images for the feature extraction network by fusing the anchor information with each of the individual images.

6. The method of claim 1 , wherein the executing of the feature extraction network comprises:

extracting an anchor local feature from the anchor image;

extracting a local feature from another image among the individual images;

extracting a global feature from the anchor local feature and the local feature; and

fusing the anchor local feature with the global feature.

7. The method of claim 1 , wherein the executing of the feature extraction network comprises:

extracting an anchor local feature from the anchor image;

extracting a local feature from another image among the individual images; and

extracting a global feature from the anchor local feature and the local feature while using the anchor local feature.

8. The method of claim 7 , wherein the executing of the global features comprises extracting the global feature from the anchor local feature and the local feature of the other image by assigning a higher weight to the anchor local feature than the local feature of the other image.

9. The method of claim 1 , wherein the generating of the restored image comprises executing an image restoration network based on the feature map.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .

11. An image restoration method comprising:

determining an anchor image based on individual images of a burst image set;

executing a feature extraction network based on the burst image set while using anchor information of the anchor image; and

generating a restored image based on a feature map corresponding to an output of the feature extraction network,

wherein the executing of the feature extraction network comprises extracting a local feature while using the anchor information for each of a plurality of layer groups of the feature extraction network, and

wherein the executing of the local feature comprises:

extracting a primary local feature from a first individual image of the individual images using a first layer group of the plurality of layer groups;

transforming the primary local feature by fusing the anchor information with the primary local feature;

extracting a secondary local feature from the transformed primary local feature using a second layer group of the plurality of layer groups; and

determining a global feature based on the secondary local feature.

12. An image restoration apparatus comprising:

one or more processors; and

a memory comprising instructions executable by the one or more processors,

wherein in response to the instructions being executed by the one or more processors, the one or more processors are configured to:

determine an anchor image based on individual images of a burst image set without aligning the individual images;

execute a feature extraction network based on the burst image set while using anchor information of the anchor image; and

generate a restored image based on a feature map corresponding to an output of the feature extraction network,

wherein the executing of the feature extraction network includes fusing the anchor information of the anchor image with features of the individual images,

wherein the one or more processors are further configured to extract a local feature wile using the anchor information for each of a plurality of layer groups of the feature extraction network, and

wherein the one or more processors, in the executing of the local feature, are further configured to:

extract a primary local feature from a first individual image of the individual images using a first layer group of the plurality of layer groups;

transform the primary local feature by fusing the anchor information with the primary local feature;

extract a secondary local feature from the transformed primary local feature using a second layer group of the plurality of layer groups; and

determine a global feature based on the secondary local feature.

13. The apparatus of claim 12 , wherein the one or more processors are further configured to:

select the anchor image among the individual images based on any one or any combination of any two or more of an image-deterioration-based selection, a time interval-based selection, and a random selection, or

generate the anchor image by applying weights to the individual images,

wherein the image-deterioration-based selection is performed based on at least one of a minimum amount of noise/blur, a maximum signal-to-noise ratio (SNR), and a highest sharpness among the individual images.

14. The apparatus of claim 12 , wherein the one or more processors are further configured to:

extract an anchor local feature from the anchor image;

extract a local feature from another image among the individual images; and

extract a global feature from the anchor local feature and the local feature of the other image while using the anchor local feature.

15. The apparatus of claim 12 , wherein the one or more processors are further configured to extract a local feature while using the anchor information for each of a plurality of layer groups of the feature extraction network.

16. An electronic apparatus comprising:

a camera configured to generate a burst image set; and

one or more processors configured to:

determine an anchor image based on individual images of the burst image set without aligning the individual images;

execute a feature extraction network based on the burst image set while using anchor information of the anchor image; and

generate a restored image based on a feature map corresponding to an output of the feature extraction network,

wherein the executing of the feature extraction network includes fusing the anchor information of the anchor image with features of the individual images,

wherein the one or more processors are further configured to extract a local feature wile using the anchor information for each of a plurality of layer groups of the feature extraction network, and

wherein the one or more processors, in the executing of the local feature, are further configured to:

extract a primary local feature from a first individual image of the individual images using a first layer group of the plurality of layer groups;

transform the primary local feature by fusing the anchor information with the primary local feature;

extract a secondary local feature from the transformed primary local feature using a second layer group of the plurality of layer groups; and

determine a global feature based on the secondary local feature.

17. The electronic apparatus of claim 16 , wherein the one or more processors are further configured to:

select the anchor image among the individual images based on any one or any combination of any two or more of an image-deterioration-based selection, a time interval-based selection, and a random selection, or

generate the anchor image by applying weights to the individual images,

wherein the image-deterioration-based selection is performed based on at least one of a minimum amount of noise/blur, a maximum signal-to-noise ratio (SNR), and a highest sharpness among the individual images.

18. The electronic apparatus of claim 16 , wherein the one or more processors are further configured to:

extract an anchor local feature from the anchor image;

extract a local feature from another image among the individual images; and

extract a global feature from the anchor local feature and the local feature of the other image while using the anchor local feature.

19. The electronic apparatus of claim 16 , wherein the one or more processors are further configured to extract a local feature while using the anchor information for each of a plurality of layer groups of the feature extraction network.