Method and apparatus with image restoration

1 . An image restoration method comprising:

convolving a first filter kernel, among a set of filter kernels, over first target data to generate a first filtering result, wherein the first target data includes first input data of layers of a neural network-based image restoration model, first auxiliary data for the first input data comprising or being derived from the first filtering result;

determining new first input data by combining the first auxiliary data with at least some of the first input data; and

generating, based on the new first input data, a restored image of an input image by inputting the new first input data to a first layer among layers of the neural network-based image restoration model, wherein the set of filter kernels are not part of any neural network,

wherein the first input data comprises at least some of the input image or at least some of output data outputted by another of the layers.

2 . The image restoration method of claim 1 , further comprising

determining second auxiliary data for output of the first layer according to the new first input data by convolving a second filter kernel among the set of filter kernels over second target data comprising the first input data and/or the output of the first layer.

3 . The image restoration method of claim 2 , further comprising

determining the new first input data by combining the first auxiliary data with the first input data; and

determining new second input data by combining the second auxiliary data with the output of the first layer according to the new first input data.

4 . The image restoration method of claim 3 , wherein

the first filter kernel and the second filter kernel comprise Gaussian filters,

the new second input data is inputted to a second layer of the layers that is closer to an output layer of the neural network-based image restoration model than the first layer, and

a size of the second filter kernel is greater than a size of the first filter kernel.

5 . The image restoration method of claim 2 ,

further comprising:

determining the new first input data by combining the first auxiliary data and the second auxiliary data with the first input data.

6 . The image restoration method of claim 2 , wherein

the determining the first auxiliary data and the determining the second auxiliary data are performed in parallel with respect to each other.

7 . The image restoration method of claim 1 , wherein the first target data is the input image,

wherein the determining the first auxiliary data comprises:

selecting the set of filter kernels according to respective different filter parameters thereof; and

determining the first auxiliary data and second auxiliary data for output of the first layer according to the new first input data by performing the convolving with the first and second filter kernels in parallel, and

wherein the generating the restored image comprises:

sequentially executing the layers of the neural network-based image restoration model.

8 . The image restoration method of claim 1 , wherein

the first filter kernel comprising a Gaussian filter for the first layer of the layers,

the first auxiliary data comprises a difference between the first target data and the first filtering result, and wherein

the determining the new input data comprises:

concatenating the first input data with the first auxiliary data.

9 . The image restoration method of claim 1 , wherein the generating the restored image comprises: generating the restored image based on a difference between an output image of the neural network-based image restoration model and the input image.

10 . The image restoration method of claim 3 , wherein

the determining the first auxiliary data and the determining the first new input data is performed through image signal processing (ISP) of a sensor that captures the input image, and

the determining the second auxiliary data and the determining the new second input data is performed through an ISP of an application processor (AP).

11 . An electronic device comprising:

a camera configured to generate an input image;

one or more processors;

storage hardware storing instructions configured to, when executed by the one or more processors, cause the electronic device to:

receive the input image,

convolve a first filter kernel, among a set of filter kernels, over first target data to generate a first filtering result, wherein the first target data includes first input data of layers of a neural network-based image restoration model, first auxiliary data for the first input data comprising or being derived from the first filtering result,

determine new first input data by combining the first auxiliary data with at least some of the first input data, and

generate, based on the new input data, a restored image of the input image by inputting the new first input data to a first layer among layers of the neural network-based image restoration model, wherein the set of filter kernels are not part of any neural network,

wherein the first input data comprises at least some of the input image or at least some output data outputted by another of the layers.

12 . The electronic device of claim 11 , wherein the instructions are further configured to cause the electronic device to, to determine the auxiliary data:

determine second auxiliary data for output of the first layer according to the new first input data by convolving a second filter kernel among the set of filter kernels over second target data comprising the first input data and/or the output of the first layer.

13 . The electronic device of claim 12 , wherein the instructions are further configured to cause the electronic device to:

determine the new first input data by integrating the first auxiliary data with the first input data, and

determine new second input data by integrating the second auxiliary data with the output of the first layer according to the new first input data.

14 . The electronic device of claim 12 , wherein the instructions are further configured to cause the electronic device to:

determine the new first input data by combining the first auxiliary data and the second auxiliary data with the first input data.

15 . The electronic device of claim 12 , wherein the instructions are further configured to cause the electronic device to determine the first auxiliary data and the second auxiliary data in parallel.

16 . A method comprising

processing an input image with an image restoration model comprising a convolutional neural network (CNN) that comprises a first convolution layer and a second convolution layer, the processing including the first convolution layer generating a first feature map as an output thereof;

filtering the input image with a filter kernel that is not part of any neural network to generate a first filtered image;

wherein the processing the image with the image restoration model includes providing both the first feature map and the first filtered image, or derivations thereof, as part of an input volume inputted to the second convolution layer; and

generating a restored image corresponding to the input image based on an output of the second convolution layer processing the input volume.

17 . The method of claim 16 , wherein the generating the restored image comprises finding a difference between an output of the CNN and the input image, and wherein the output of the CNN is based on the output of the second convolution layer.

18 . The method of claim 16 , wherein providing the first filtered image increases a number of channels that are input to the second convolution layer.