Electronic device for performing image deblurring and controlling method of electronic device
An image processing device includes an encoder configured to receive a blurry image and generate a global feature map of the image, a merging unit configured to merge the global feature map and blur information, a decoder configured to generate a feature tensor and weight tensors, a recurrent refinement module configured to perform recurrent feature filtering, and an image reconstruction module configured to reconstruct a deblurred image where the image processing device is configured to estimate an image global shift and to activate or deactivate the recurrent refinement module based on the estimation.
1. An electronic device comprising:
a memory configured to store at least one neural network model to perform deblurring of an image; and
a processor configured to:
detect, using a motion sensor, blur information characterizing motion of a camera at a time of capturing an image,
input the image comprising a blurred area and the blur information to a first neural network model to obtain a first feature information corresponding to the image and to obtain a weight value information corresponding to the first feature information,
obtain a global shift information indicating whether an image shift included in the image is local in certain areas of the image or global throughout a scene in the image
based on the global shift information indicating that the image shift included in the image is local in the certain areas of the image, identify the scene included in the image as a dynamic scene, obtain a second feature information in which the first feature information is filtered by performing a recurrent filtering process based on the first feature information and the weight value information through a second neural network model and obtain an image in which the blurred area is deblurred by inputting the second feature information to a third neural network model, and
based on the global shift information indicating that the image shift included in the image is global throughout the scene in the image, identify the scene included in the image as a static scene, and obtain an image in which the blurred area is deblurred by inputting the first feature information to the third neural network model without activating the second neural network model.
2. The electronic device according to claim 1 , wherein the first neural network model is configured to obtain a global feature map corresponding to the image and obtain the first feature information by using the global feature map and the blur information.
3. The electronic device according to claim 1 , wherein the second neural network model comprises a plurality of recurrent neural network (RNN) layers successively processing the first feature information, and
each of the plurality of RNN layers performs the recurrent filtering process based on image sweeps in different directions from one another.
4. A non-transitory computer readable recording medium comprising a program executing a controlling method of an electronic device, the controlling method comprises:
detecting, using a motion sensor, blur information characterizing motion of a camera at a time of capturing an image;
inputting the image comprising a blurred area and the blur information to a first neural network model to obtain a first feature information corresponding to the image and to obtain a weight value information corresponding to the feature information;
obtaining a global shift information indicating whether an image shift included in the image is local in certain areas of the image or global throughout a scene in the image;
based on the global shift information indicating that the image shift included in the image is local in the certain areas of the image, identifying the scene included in the image as a dynamic scene, obtaining a second feature information in which the first feature information is filtered by performing a recurrent filtering process based on the first feature information and the weight value information through a second neural network model and obtaining an image in which the blurred area is deblurred by inputting the second feature information to a third neural network model; and
based on the global shift information indicating that the image shift included in the image is global throughout the scene in the image, identifying the scene included in the image as a static scene, and obtaining an image in which the blurred area is deblurred by inputting the first feature information to the third neural network model without activating the second neural network model.