Conditional kernel prediction network and adaptive depth prediction for image and video processing
Embodiments are generally directed to a Conditional Kernel Prediction Network (CKPN) for image and video de-noising and other related image and video processing applications. Disclosed is an embodiment of a method for de-noising an image or video frame by a convolutional neural network implemented on a compute engine, the image including a plurality of pixels, the method comprising: for each of the plurality of pixels of the image, generating a convolutional kernel having a plurality of kernel weights for the pixel, the plurality of kernel weights respectively corresponding to pixels within a region surrounding the pixel; adjusting the plurality of kernel weights of the convolutional kernel for the pixel based on convolutional kernels generated respectively for the corresponding pixels within the region surrounding the pixel; and filtering the pixel with the adjusted plurality of kernel weights and pixel values of the corresponding pixels within the region surrounding the pixel to obtain a de-noised pixel.
1 . An apparatus comprising:
processor circuitry coupled to a memory, the processor circuitry to:
generate convolutional kernels having kernel weights corresponding to pixels associated with an image, wherein the pixels are within a region surrounding a pixel associated with the image;
adjust a kernel weight of a convolutional kernel corresponding to the pixel based on the convolutional kernels generated respectively for the pixels, wherein the kernel weight is further adjusted based on one or more of a position of the pixel relative to the pixels within the region surrounding the pixel associated with the image or a correlation between the convolutional kernel corresponding to the pixel and the convolutional kernels generated for the pixels within the region surrounding the pixel;
obtain a de-noised pixel by filtering the pixel based on the adjusted kernel weight and a pixel value associated with the pixel and further based on the kernel weights and pixel values associated with the pixels within the region surrounding the pixel; and
infer the convolutional kernel corresponding to the pixel based on a target depth of a convolutional neural network, wherein the target depth is individually determined for the pixels associated with the image by the convolutional neural network during inference, wherein the convolutional neural network is trained with a loss calculated based on a randomly sampled target depth and predicted target depths during iterations associated with training, wherein the target depths are predicted based on adaptive depth prediction to dynamically select depth values for the pixels during training based on image characteristics, and wherein the adaptive depth prediction adjusts the target depth for the pixels to optimize kernel generation for local neighborhood characteristics associated with the pixels.
2 . The apparatus of claim 1 , wherein the processor circuitry is further to:
determine the target depth of the convolutional neural network for generating the convolutional kernels for the pixels, wherein the target depth being less than or equal to a full depth associated with the convolutional neural network.
3 . The apparatus of claim 1 , wherein the processor circuitry comprises one or more of graphics processor circuitry or application processor circuitry.
4 . A method comprising:
generating, by a processor of a computing device, convolutional kernels having kernel weights corresponding to pixels associated with an image, wherein the pixels are within a region surrounding a pixel associated with the image;
adjusting a kernel weight of a convolutional kernel corresponding to the pixel based on the convolutional kernels generated respectively for the pixels, wherein the kernel weight is further adjusted based on one or more of a position of the pixel relative to the pixels within the region surrounding the pixel associated with the image or a correlation between the convolutional kernel corresponding to the pixel and the convolutional kernels generated for the pixels within the region surrounding the pixel;
obtaining a de-noised pixel by filtering the pixel based on the adjusted kernel weight and a pixel value associated with the pixel and further based on the kernel weights and pixel values associated with the pixels within the region surrounding the pixel; and
inferring the convolutional kernel corresponding to the pixel based on a target depth of a convolutional neural network, wherein the target depth is individually determined for the pixels associated with the image by the convolutional neural network during inference, wherein the convolutional neural network is trained with a loss calculated based on a randomly sampled target depth and predicted target depths during iterations associated with training, wherein the target depths are predicted based on adaptive depth prediction to dynamically select depth values for the pixels during training based on image characteristics, and wherein the adaptive depth prediction adjusts the target depth for the pixels to optimize kernel generation for local neighborhood characteristics associated with the pixels.
5 . The method of claim 4 , further comprising:
determine a target depth of a convolutional neural network for generating the convolutional kernels for the pixels, wherein the target depth being less than or equal to a full depth associated with the convolutional neural network.
6 . The method of claim 4 , wherein the processor is coupled to a memory, the processor comprises one or more of a graphics processor or an application processor.
7 . At least one non-transitory computer-readable medium having stored thereon instructions which, when executed, cause a computing device to perform operations comprising:
generating convolutional kernels having kernel weights corresponding to pixels associated with an image, wherein the pixels are within a region surrounding a pixel associated with the image;
adjusting a kernel weight of a convolutional kernel corresponding to the pixel based on the convolutional kernels generated respectively for the pixels, wherein the kernel weight is further adjusted based on one or more of a position of the pixel relative to the pixels within the region surrounding the pixel associated with the image or a correlation between the convolutional kernel corresponding to the pixel and the convolutional kernels generated for the pixels within the region surrounding the pixel;
obtaining a de-noised pixel by filtering the pixel based on the adjusted kernel weight and a pixel value associated with the pixel and further based on the kernel weights and pixel values associated with the pixels within the region surrounding the pixel; and
inferring the convolutional kernel corresponding to the pixel based on a target depth of a convolutional neural network, wherein the target depth is individually determined for the pixels associated with the image by the convolutional neural network during inference, wherein the convolutional neural network is trained with a loss calculated based on a randomly sampled target depth and predicted target depths during iterations associated with training, wherein the target depths are predicted based on adaptive depth prediction to dynamically select depth values for the pixels during training based on image characteristics, and wherein the adaptive depth prediction adjusts the target depth for the pixels to optimize kernel generation for local neighborhood characteristics associated with the pixels.
8 . The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise:
determine a target depth of a convolutional neural network for generating the convolutional kernels for the pixels, wherein the target depth being less than or equal to a full depth associated with the convolutional neural network.
9 . The non-transitory computer-readable medium of claim 7 , wherein the computing device comprises processor circuitry coupled to a memory, the processor circuitry includes one or more of graphics processor circuitry or an application processor circuitry.