Method and apparatus for image restoration based on burst image
A method and apparatus for image restoration based on burst images. The method includes generating a plurality of feature representations corresponding to individual images of a burst image set by encoding the individual images, determining a reference feature representation from among the plurality of feature representations, determining a first comparison pair including the reference feature representation and a first feature representation of the plurality of feature representations, generating a first motion-embedding feature representation of the first comparison pair based on a similarity score map of the reference feature representation and the first feature representation, generating a fusion result by fusing a plurality of motion-embedding feature representations including the first motion-embedding feature representation, and generating at least one restored image by decoding the fusion result.
1 . A processor-implemented image restoration method, comprising:
generating a plurality of feature representations corresponding to a plurality of individual images of a burst image set where the plurality of individual images was continuously captured, the plurality of feature representations were generated by encoding the individual images;
determining a feature representation from among the plurality of feature representations as a reference feature representation;
determining a first comparison pair comprising the reference feature representation and a first feature representation of the plurality of feature representations;
generating, based on a similarity score map of the reference feature representation and the first feature representation, a first motion-embedding feature representation of the first comparison pair;
generating a fusion result by fusing a plurality of motion-embedding feature representations comprising the first motion-embedding feature representation; and
generating at least one restored image by decoding the fusion result.
2 . The processor-implemented image restoration method of claim 1 , wherein the generating of the first motion-embedding feature representation comprises:
determining a first feature vector of a first point of the reference feature representation, and a first block of interest centered at a correspondence point of the first point in the first feature representation;
generating a first similarity score map corresponding to the first point based on similarities between the first feature vector and channel vectors of the first block of interest;
generating a first motion-aware kernel corresponding to the first point based on the first similarity score map;
generating a first motion-embedding vector corresponding to the first point based on an operation between the first block of interest and the first motion-aware kernel; and
generating the first motion-embedding feature representation based on motion-embedding vectors comprising the first motion-embedding vector.
3 . The processor-implemented image restoration method of claim 2 , wherein the generating of the first motion-embedding feature representation comprises:
determining a second feature vector of a second point of the reference feature representation, and a second block of interest centered at a correspondence point of the second point in the first feature representation; and
generating a second motion-embedding vector corresponding to the second point based on the second block of interest,
wherein the generating of the first motion-embedding feature representation comprises generating the first motion-embedding feature representation based on the motion-embedding vectors comprising the first motion-embedding vector and the second motion-embedding vector.
4 . The processor-implemented image restoration method of claim 2 , wherein the generating of the first motion-aware kernel comprises:
determining a first motion information vector corresponding to the first similarity score map;
generating a first context information vector based on an operation between the first block of interest and the first similarity score map; and
generating the first motion-aware kernel by fusing the first motion information vector and the first context information vector.
5 . The processor-implemented image restoration method of claim 4 , wherein the generating of the first motion-aware kernel by fusing the first motion information vector and the first context information vector comprises:
generating a first channel vector corresponding to the first motion information vector using a first fully connected network;
generating a second channel vector corresponding to the first context information vector using a second fully connected network; and
generating the first motion-aware kernel by fusing the first channel vector and the second channel vector.
6 . The processor-implemented image restoration method of claim 1 , wherein
the generating of the feature representations comprises encoding the individual images using an encoding model comprising at least one convolutional layer, and
the generating of the at least one restored image comprises decoding the fusion result using a decoding model comprising at least one up-convolutional layer.
7 . The processor-implemented image restoration method of claim 1 , further comprising:
selecting an anchor image from among the individual images,
wherein the determining of the reference feature representation comprises determining a feature representation of the anchor image from among the feature representations to be the reference feature representation.
8 . The processor-implemented image restoration method of claim 7 , wherein the selecting of the anchor image comprises selecting the anchor image from among the individual images based on quality.
9 . The processor-implemented image restoration method of claim 1 , wherein
the individual images comprise a first individual image and a second individual image, and
the at least one restored image comprises a first restored image of the first individual image and a second restored image of the second individual image,
wherein the determining of the reference feature representation comprises:
determining a feature representation of the first individual image from among the feature representations to be the reference feature representation, in case of restoring the first individual image as the first restored image; and
determining a feature representation of the second individual image from among the feature representations to be the reference feature representation, in case of restoring the second individual image as the second restored image.
10 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image restoration method of claim 1 .
11 . An image restoration apparatus, comprising:
a processor; and
a memory configured to store instructions executable by the processor,
wherein in response to the instructions being executed by the processor, the processor is configured to:
generate a plurality of feature representations corresponding to a plurality of individual images of a burst image set where the plurality of individual images was continuously captured, the plurality of feature representations were generated by encoding the individual images,
determine a reference feature representation from among the plurality of feature representations,
determine a first comparison pair comprising the reference feature representation and a first feature representation of the plurality of feature representations,
generate, based on a similarity score map of the reference feature representation and the first feature representation, a first motion-embedding feature representation of the first comparison pair,
generate a fusion result by fusing a plurality of motion-embedding feature representations comprising the first motion-embedding feature representation, and
generate at least one restored image by decoding the fusion result.
12 . The image restoration apparatus of claim 11 , wherein the processor is further configured to:
determine a first feature vector of a first point of the reference feature representation, and a first block of interest centered at a correspondence point of the first point in the first feature representation,
generate a first similarity score map corresponding to the first point based on similarities between the first feature vector and channel vectors of the first block of interest,
generate a first motion-aware kernel corresponding to the first point based on the first similarity score map,
generate a first motion-embedding vector corresponding to the first point based on an operation between the first block of interest and the first motion-aware kernel, and
generate the first motion-embedding feature representation based on motion-embedding vectors comprising the first motion-embedding vector.
13 . The image restoration apparatus of claim 12 , wherein the processor is further configured to:
determine a second feature vector of a second point of the reference feature representation, and a second block of interest centered at a correspondence point of the second point in the first feature representation,
generate a second motion-embedding vector corresponding to the second point based on the second block of interest, and
generate the first motion-embedding feature representation based on the motion-embedding vectors comprising the first motion-embedding vector and the second motion-embedding vector.
14 . The image restoration apparatus of claim 12 , wherein the processor is further configured to:
determine a first motion information vector corresponding to the first similarity score map,
generate a first context information vector based on an operation between the first block of interest and the first similarity score map, and
generate the first motion-aware kernel by fusing the first motion information vector and the first context information vector.
15 . The image restoration apparatus of claim 14 , wherein the processor is further configured to:
generate a first channel vector corresponding to the first motion information vector using a first fully connected network,
generate a second channel vector corresponding to the first context information vector using a second fully connected network, and
generate the first motion-aware kernel by fusing the first channel vector and the second channel vector.
16 . The image restoration apparatus of claim 11 , wherein the processor is further configured to:
select an anchor image from among the individual images, and
determine a feature representation of the anchor image from among the feature representations to be the reference feature representation.
17 . The image restoration apparatus of claim 11 , wherein
the individual images comprise a first individual image and a second individual image, and
the at least one restored image comprises a first restored image of the first individual image and a second restored image of the second individual image,
wherein the processor is further configured to:
determine a feature representation of the first individual image from among the feature representations to be the reference feature representation, in case of restoring the first individual image as the first restored image, and
determine a feature representation of the second individual image from among the feature representations to be the reference feature representation, in case of restoring the second individual image as the second restored image.
18 . An electronic device, comprising:
a camera configured to generate a burst image set comprising a plurality of individual images where the plurality of individual images was continuously captured; and
a processor configured to:
generate a plurality of feature representations corresponding to the individual images by encoding the individual images,
determine a reference feature representation from among the plurality of feature representations,
determine a first comparison pair comprising the reference feature representation and a first feature representation of the plurality of feature representations,
generate, based on a similarity score map of the reference feature representation and the first feature representation, a first motion-embedding feature representation of the first comparison pair,
generate a fusion result by fusing a plurality of motion-embedding feature representations comprising the first motion-embedding feature representation, and
generate at least one restored image by decoding the fusion result.
19 . The electronic device of claim 18 , wherein the processor is further configured to:
determine a first feature vector of a first point of the reference feature representation, and a first block of interest centered at a correspondence point of the first point in the first feature representation,
generate a first similarity score map corresponding to the first point based on similarities between the first feature vector and channel vectors of the first block of interest,
generate a first motion-aware kernel corresponding to the first point based on the first similarity score map,
generate a first motion-embedding vector corresponding to the first point based on an operation between the first block of interest and the first motion-aware kernel, and
generate the first motion-embedding feature representation based on motion-embedding vectors comprising the first motion-embedding vector.
20 . The electronic device of claim 19 , wherein the processor is further configured to:
determine a first motion information vector corresponding to the first similarity score map,
generate a first context information vector based on an operation between the first block of interest and the first similarity score map, and
generate the first motion-aware kernel by fusing the first motion information vector and the first context information vector.
21 . An electronic device, comprising:
one or more processors configured to:
acquire a plurality of individual images included in an image burst set where the plurality of individual images was continuously captured;
calculate a correlation between the individual images to generate a kernel;
generate a feature representation of the individual images by applying a convolution operation to the individual images using the kernel; and
enhance the individual images based on the feature representation and output an enhanced image burst set including the enhanced individual images.
22 . The electronic device of claim 21 , wherein the one or more processors are configured to:
for the calculating of the correlation between the individual images to generate the kernel,
generate a plurality of feature representations corresponding to the individual images by encoding the individual images,
determine a reference feature representation from among the plurality of feature representations, and
determine a first comparison pair comprising the reference feature representation and a first feature representation of the plurality of feature representations;
for the generating of the feature representation, generate, as the feature representation, a first motion-embedding feature representation of the first comparison pair based on a similarity score map of the reference feature representation and the first feature representation;
generate a fusion result by fusing a plurality of motion-embedding feature representations comprising the first motion-embedding feature representation; and
for the enhancing of the individual images, generate at least one restored image by decoding the fusion result.