Optimization of adaptive convolutions for video frame interpolation

1. A computer-implemented method comprising:

receiving a pair of input images from a digital video;

determining, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image;

convolving a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images;

generating a first filtered mask by convolving the first set of spatially varying kernels with a first singular mask;

generating a second filtered mask by convolving the second set of spatially varying kernels with a second singular mask; and

generating the output image by dividing the filtered images by the first and second filtered masks.

2. The computer-implemented method of claim 1 , wherein determining, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image, further comprises:

estimating, by the neural network, kernel coefficients for a plurality of pairs of one-dimensional kernels; and

convolving each pair of one-dimensional kernels to generate the plurality of spatially varying kernels.

3. The computer-implemented method of claim 1 , further comprising:

transforming the first input image and the second input image to generate a pair of transformed input images;

determining, by the neural network, a second plurality of spatially varying kernels corresponding to a pixel of a second output image;

convolving a first set of spatially varying kernels from the second plurality of spatially varying kernels with a first transformed input image from the pair of transformed input images and a second set of spatially varying kernels from the second plurality of spatially varying kernels with a second transformed input image from the pair of transformed input images to generate second filtered images;

generating the second output image by performing kernel normalization on the second filtered images; and

combining the output image and the second output image.

4. The computer-implemented method of claim 1 , wherein convolving a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate a filtered image, further comprises:

performing adaptive separable convolution on the first input image using the first set of spatially varying kernels; and

performing adaptive separable convolution on the second input image using the second set of spatially varying kernels.

5. The computer-implemented method of claim 1 , wherein the neural network is trained by:

obtaining a pair of training images and a ground truth image;

generating an output training image based on the pair of training images using the neural network;

generating a feature representation of the output training image and the ground truth image using a pretrained neural network; and

minimizing a contextual loss function calculated from the output training image, the ground truth image, the feature representation of the output training image, and the feature representation of the ground truth image.

6. The computer-implemented method of claim 1 , wherein convolving a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images, further comprises:

generating a padded first input image and a padded second input image; and

convolving the first padded input image with the first set of spatially varying kernels and the second padded input image with the second set of spatially varying kernels to generate the filtered images.

7. The computer-implemented method of claim 1 , wherein the plurality of spatially varying kernels are filter kernels for video image interpolation and wherein the output image is an output intermediate image.

8. The computer-implemented method of claim 1 , wherein the plurality of spatially varying kernels are denoising kernels or deblurring kernels.

9. A non-transitory computer-readable storage medium including instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:

receive a pair of input images from a digital video;

determine, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image;

convolve a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images;

generate a first filtered mask by convolving the first set of spatially varying kernels with a first singular mask;

generate a second filtered mask by convolving the second set of spatially varying kernels with a second singular mask; and

generate the output image by dividing the filtered images by the first and second filtered masks.

10. The non-transitory computer-readable storage medium of claim 9 , wherein to determine, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image, the instructions, when executed, further cause the at least one processor to:

estimate, by the neural network, kernel coefficients for a plurality of pairs of one-dimensional kernels; and

convolve each pair of one-dimensional kernels to generate the plurality of spatially varying kernels.

11. The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the at least one processor to:

transform the first input image and the second input image to generate a pair of transformed input images;

determine, by the neural network, a second plurality of spatially varying kernels corresponding to a pixel of a second output image;

convolve a first set of spatially varying kernels from the second plurality of spatially varying kernels with a first transformed input image from the pair of transformed input images and a second set of spatially varying kernels from the second plurality of spatially varying kernels with a second transformed input image from the pair of transformed input images to generate second filtered images;

generate the second output image by performing kernel normalization on the second filtered images; and

combine the output image and the second output image.

12. The non-transitory computer-readable storage medium of claim 9 , wherein to convolve a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate a filtered image, the instructions, when executed, further cause the at least one processor to:

perform adaptive separable convolution on the first input image using the first set of spatially varying kernels; and

perform adaptive separable convolution on the second input image using the second set of spatially varying kernels.

13. The non-transitory computer-readable storage medium of claim 9 , wherein the neural network is trained by:

obtaining a pair of training images and a ground truth image;

generating an output training image based on the pair of training images using the neural network;

generating a feature representation of the output training image and the ground truth image using a pretrained neural network; and

minimizing a contextual loss function calculated from the output training image, the ground truth image, the feature representation of the output training image, and the feature representation of the ground truth image.

14. The non-transitory computer-readable storage medium of claim 9 , wherein to convolve a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images, the instructions, when executed, further cause the at least one processor to:

generate a padded first input image and a padded second input image; and

convolve the first padded input image with the first set of spatially varying kernels and the second padded input image with the second set of spatially varying kernels to generate the filtered images.

15. A system comprising:

one or more electronic devices including at least one processor and memory, the one or more electronic devices implementing an image processing system including instructions which, when executed by the at least one processor, cause the image processing system to:

receive a pair of input images from a digital video;

determine, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image;

convolve a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images;

generate a first filtered mask by convolving the first set of spatially varying kernels with a first singular mask;

generate a second filtered mask by convolving the second set of spatially varying kernels with a second singular mask; and

generate the output image by dividing the filtered images by the first and second filtered masks.

16. The system of claim 15 , wherein to determine, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image, the instructions, when executed, further cause the image processing system to:

estimate, by the neural network, kernel coefficients for a plurality of pairs of one-dimensional kernels; and

convolve each pair of one-dimensional kernels to generate the plurality of spatially varying kernels.

17. The system of claim 15 , wherein the instructions, when executed, further cause the image processing system to:

transform the first input image and the second input image to generate a pair of transformed input images;

determine, by the neural network, a second plurality of spatially varying kernels corresponding to a pixel of a second output image;

convolve a first set of spatially varying kernels from the second plurality of spatially varying kernels with a first transformed input image from the pair of transformed input images and a second set of spatially varying kernels from the second plurality of spatially varying kernels with a second transformed input image from the pair of transformed input images to generate second filtered images;

generate the second output image by performing kernel normalization on the second filtered images; and

combine the output image and the second output image.

18. The system of claim 15 , wherein to convolve a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate a filtered image, the instructions, when executed, further cause the image processing system to:

perform adaptive separable convolution on the first input image using the first set of spatially varying kernels; and

perform adaptive separable convolution on the second input image using the second set of spatially varying kernels.