Deblurring motion in videos

1. A computer-implemented method comprising:

receiving an input frame from a digital video;

providing a combined input to a neural network, the combined input including the input frame, a previous input frame, and a corresponding previous output frame;

extracting a plurality of combined features of the combined input using an encoder network;

determining, using the neural network, a plurality of spatial alignment kernels and a plurality of deblur kernels each corresponding to a feature of the plurality of combined features, wherein the plurality of spatial alignment kernels include different sizes of spatial alignment kernels and wherein the plurality of deblur kernels include different sizes of deblur kernels;

padding the plurality of deblur kernels such that each of the plurality of deblur kernels is of equal size;

averaging the plurality of padded deblur kernels;

convolving the average of the plurality of padded deblur kernels with a plurality of features of the input frame to obtain a first convolved result;

generating, by the neural network, a plurality of output features for the input frame using the plurality of spatial alignment kernels and the first convolved result; and

generating a deblurred output frame from the plurality of output features using a decoder network.

2. The computer-implemented method of claim 1 wherein the plurality of spatial alignment kernels include different sizes of spatial alignment kernels and wherein the plurality of deblur kernels include different sizes of deblur kernels, further comprises:

predicting, by the neural network, a plurality of pairs of one-dimensional deblur filters corresponding to the plurality of deblur kernels for each feature from the plurality of combined features; and

predicting, by the neural network, a plurality of pairs of one-dimensional spatial alignment filters corresponding to the plurality of spatial alignment kernels for each feature from the plurality of combined features.

3. The computer-implemented method of claim 2 , wherein generating, by the neural network, a plurality of output features for the input frame using the plurality of spatial alignment kernels and the plurality of deblur kernels, further comprises:

convolving the plurality of spatial alignment kernels with a plurality of features of the previous output frame to obtain a second convolved result; and

concatenating the first convolved result with the second convolved result to generate the plurality of output features.

4. The computer-implemented method of claim 3 , wherein convolving the plurality of spatial alignment kernels with a plurality of features of the previous output frame to obtain a second convolved result, further comprises:

padding the plurality of spatial alignment kernels such that each of the plurality of spatial alignment kernels is of equal size;

averaging the plurality of padded spatial alignment kernels; and

convolving the average of the plurality of padded spatial alignment kernels with the plurality of features of the previous output frame to obtain the second convolved result.

5. The computer-implemented method of claim 3 , further comprising:

downsampling the plurality of output features to generate a plurality of downsampled output features;

storing the plurality of downsampled output features for processing a next frame; and

storing the plurality of combined features for processing the next frame.

6. The computer-implemented method of claim 1 , wherein the neural network is a multi kernel adaptive separable convolution network (MKASCN).

7. The computer-implemented method of claim 1 , further comprising:

sequentially processing each frame of the digital video to obtain a deblurred output video.

8. A non-transitory computer-readable storage medium including instructions stored there on which, when executed by at least one processor, cause the at least one processor to:

receive an input frame from a digital video;

provide a combined input to a neural network, the combined input including the input frame, a previous input frame, and a corresponding previous output frame;

extract a plurality of combined features of the combined input using an encoder network;

determine, using the neural network, a plurality of spatial alignment kernels and a plurality of deblur kernels each corresponding to a feature of the plurality of combined features, wherein the plurality of spatial alignment kernels include different sizes of spatial alignment kernels and wherein the plurality of deblur kernels include different sizes of deblur kernels;

pad the plurality of deblur kernels such that each of the plurality of deblur kernels is of equal size;

average the plurality of padded deblur kernels;

convolve the average of the plurality of padded deblur kernels with a plurality of features of the input frame to obtain a first convolved result;

generate, by the neural network, a plurality of output features for the input frame using the plurality of spatial alignment kernels and the first convolved result; and

generate a deblurred output frame from the plurality of output features using a decoder network.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the plurality of spatial alignment kernels include different sizes of spatial alignment kernels and wherein the plurality of deblur kernels include different sizes of deblur kernels, the instructions, when executed, further cause the at least one processor to:

predict, by the neural network, a plurality of pairs of one-dimensional deblur filters corresponding to the plurality of deblur kernels for each feature from the plurality of combined features; and

predict, by the neural network, a plurality of pairs of one-dimensional spatial alignment filters corresponding to the plurality of spatial alignment kernels for each feature from the plurality of combined features.

10. The non-transitory computer-readable storage medium of claim 9 , wherein to generate, by the neural network, a plurality of output features for the input frame using the plurality of spatial alignment kernels and the plurality of deblur kernels, the instructions, when executed, further cause the at least one processor to:

convolve the plurality of spatial alignment kernels with a plurality of features of the previous output frame to obtain a second convolved result; and

concatenate the first convolved result with the second convolved result to generate the plurality of output features.

11. The non-transitory computer-readable storage medium of claim 10 , wherein to convolving the plurality of spatial alignment kernels with a plurality of features of the previous output frame to obtain a second convolved result, the instructions, when executed, further cause the at least one processor to:

pad the plurality of spatial alignment kernels such that each of the plurality of spatial alignment kernels is of equal size;

average the plurality of padded spatial alignment kernels; and

convolve the average of the plurality of padded spatial alignment kernels with the plurality of features of the previous output frame to obtain the second convolved result.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions, when executed, further cause the at least one processor to:

downsample the plurality of output features to generate a plurality of downsampled output features;

store the plurality of downsampled output features for processing a next frame; and

store the plurality of combined features for processing the next frame.

13. The non-transitory computer-readable storage medium of claim 8 , wherein the neural network is a multi kernel adaptive separable convolution network (MKASCN).

14. The non-transitory computer-readable storage medium of claim 8 , wherein the instructions, when executed, further cause the at least one processor to:

sequentially process each frame of the digital video to obtain a deblurred output video.

15. A method comprising:

receiving a request to deblur a digital video, the request including at least a reference to the digital video;

obtaining the digital video;

processing a plurality of frames of the digital video in sequence by a multi kernel adaptive separable convolution network (MKASCN) to generate a deblurred and spatially aligned output video spatially aligned output video, wherein processing the plurality of frames includes:

providing a combined input to the MKASCN, the combined input including an input frame from the digital video, a previous input frame, and a corresponding previous output frame;

extracting a plurality of combined features of the combined input using an encoder network;

determining, using the MKASCN, a plurality of spatial alignment kernels corresponding to a feature of the plurality of combined features, and a plurality of deblur kernels corresponding to a feature of the plurality of combined features,

padding the plurality of spatial alignment kernels such that each of the plurality of spatial alignment kernels is of equal size,

averaging the plurality of padded spatial alignment kernels,

convolving the average of the plurality of padded spatial alignment kernels with a plurality of features of the previous output frame to obtain a first convolved result, and

generating a deblurred output frame corresponding to a frame of the plurality of frames using the plurality of deblur kernels and the first convolved result; and

returning the deblurred and spatially aligned output video.

16. The method of claim 15 , further comprises:

generating, by the MKASCN, a plurality of output features for the input frame using the plurality of spatial alignment kernels and the plurality of deblur kernels;

generating a deblurred output frame from the plurality of output features using a decoder network; and

sequentially processing each frame of the digital video to obtain the deblurred and spatially aligned output video.

17. The method of claim 16 , further comprises:

predicting, by the MKASCN, a plurality of pairs of one-dimensional deblur filters corresponding to the plurality of deblur kernels for each feature from the plurality of combined features; and

predicting, by the MKASCN, a plurality of pairs of one-dimensional spatial alignment filters corresponding to the plurality of spatial alignment kernels for each feature from the plurality of combined features.

18. The method of claim 17 , wherein generating, by the MKASCN, a plurality of output features for the input frame using the plurality of spatial alignment kernels and the plurality of deblur kernels, further comprises:

convolving the plurality of deblur kernels with the plurality of features of the input frame to obtain a second convolved result; and

concatenating the first convolved result with the second convolved result to generate the plurality of output features.