Unsupervised learning approach for video deblurring
An image processing method extracts consecutive input blurry frames from a video, and generates sharp frames corresponding to the input blurry frames. An optical flow is determined between the sharp frames, and the optical flow is used to compute a per-pixel blur kernel. The blur kernel is used to reblur each of the sharp frames into a corresponding re-blurred frame. The re-blurred frame is used to fine-tune the deblur network by minimizing the distance between the re-blurred frame and the input blurry frame.
1. An image processing method comprising:
extracting a plurality of consecutive input blurry frames from a video;
generating a plurality of sharp frames, each of the sharp frames corresponding to one of the input blurry frames;
determining an optical flow between the sharp frames;
applying the optical flow to compute a per-pixel blur kernel; and
applying the blur kernel to reblur each of the sharp frames into a corresponding re-blurred frame.
2. The image processing method of claim 1 , wherein:
generating the sharp frames is performed with a first neural network; and
generating the optical flow is performed with a second neural network.
3. The image processing method of claim 2 , wherein the first neural network is trained using the re-blurred frames.
4. The image processing method of claim 3 , wherein the training of the first neural network minimizes a distance between each of the input blurry frames and the corresponding re-blurred frame.
5. The image processing method of claim 3 , wherein the training of the first neural network comprises backpropagation of weights derived from both of a determined supervised loss in the re-blurred frames and a determined unsupervised loss in the re-blurred frames.
6. The image processing method of claim 2 , wherein the first neural network is one of DVD and DeblurGAN.
7. The image processing method of claim 2 , wherein the second neural network is a FlowNetS network.
8. The image processing method of claim 1 , wherein the reblur is performed utilizing a precomputed lookup table that maps the optical flow to the blur kernel using bilinear interpolation.
9. The image processing method of claim 1 , wherein the reblur is performed using a per-pixel convolution with the blur kernel.
10. The image processing method of claim 9 , wherein the blur kernel comprises a pair of piecewise linear line segments per pixel.
11. An image processing apparatus comprising:
a graphics processing unit; and
logic to configure the graphics processing unit to:
extract a plurality of consecutive input blurry frames from a video;
generate a plurality of sharp frames, each of the sharp frames corresponding to one of the input blurry frames;
determine an optical flow between the sharp frames;
apply the optical flow to compute a per-pixel blur kernel; and
apply the blur kernel to reblur each of the sharp frames into a corresponding re-blurred frame.
12. The apparatus of claim 11 , wherein the logic further configures the graphics processing unit to:
generate the sharp frames with a first neural network; and
generate the optical flow with a second neural network.
13. The apparatus of claim 12 , further comprising logic to train the first neural network using the re-blurred frames.
14. The apparatus of claim 13 , wherein the training of the first neural network minimizes a distance between each of the input blurry frames and the corresponding re-blurred frame.
15. The apparatus of claim 13 , wherein the training of the first neural network comprises backpropagation of weights derived from both of a determined supervised loss in the re-blurred frames and a determined unsupervised loss in the re-blurred frames.
16. The apparatus of claim 12 , wherein the first neural network is one of DVD and DeblurGAN.
17. The apparatus of claim 12 , wherein the second neural network is a FlowNetS network.
18. The apparatus of claim 11 , wherein the reblur is performed utilizing a precomputed lookup table that maps the optical flow to the blur kernel using bilinear interpolation.
19. The apparatus of claim 11 , wherein the reblur is performed using a per-pixel convolution with the blur kernel.
20. The apparatus of claim 19 , wherein the blur kernel comprises a pair of piecewise linear line segments per pixel.