Demosaicing module for an image signal processing pipeline
A demosaicing module for an image signal processing pipeline comprises a neural network comprising a plurality of branches, each branch comprising a plurality of successively executable convolutional layers, one convolutional layer of each branch being configured to receive a plurality of image planes acquired from a colour filter array, CFA, image sensor, each image plane having a resolution equal to the image sensor, with only one image plane having a non-blank pixel value at any pixel location and with pixel information for each colour plane maintained in a spatial relationship corresponding to the pixel locations for each colour of the colour filter array. The convolutional layers of each branch are configured to combine the image plane information into a demosaiced image plane in a given image space.
1 . An image signal-processing pipeline (ISP) comprising a demosaicing module, the demosaicing module comprising a neural network comprising a plurality of branches, each branch comprising a plurality of successively executable convolutional layers, one convolutional layer of each branch being configured to receive a plurality of image planes acquired from a color filter array (CFA) image sensor, each image plane having a resolution equal to the image sensor, with only one image plane having a non-blank pixel value at any pixel location and with pixel information for each color plane maintained in a spatial relationship corresponding to pixel locations for each color of the CFA, the convolutional layers of each branch being configured to combine image plane information into a demosaiced image plane in a given output image space,
wherein the ISP further comprises a lens shading correction module configured to compensate for brightness variation in the image channels acquired from the CFA image sensor before passing corrected image information to the demosaicing module, and
wherein the CFA comprises one of an: RGB-IR, RGB-W or hyperspectral filter array.
2 . The ISP of claim 1 , wherein the CFA comprises a Bayer filter and where image information acquired from the image sensor comprises Red, Green and Blue image channels.
3 . The ISP of claim 2 , wherein the neural network comprises three branches, each branch being configured to produce a respective image plane for an output image in one of: RGB, YCC, YUV, LAB or XYZ color spaces.
4 . The ISP of claim 3 wherein the output image space is RGB and wherein the structure of each of the three branches is identical.
5 . The ISP of claim 3 wherein the output image space is YUV and wherein the respective branches configured to provide demosaiced U and V image planes include one or more downsampling layers.
6 . The ISP of claim 1 wherein each branch comprises one or more residual blocks, each block comprising one or more convolutional layers followed by a respective activation function.
7 . The ISP of claim 1 wherein at least one convolution layer of each branch comprises at least one 5×5 convolution kernel.
8 . The ISP of claim 1 wherein the neural network is trained based on input image information which comprises one or more of: blur, noise, aberrations or defects relative to ground truth information so that the demosaicing module is configured to correct input information accordingly during demosaicing.
9 . The ISP of claim 1 wherein one or more of the branches comprises an upsampling layer to increase the spatial resolution of the demosaiced image information relative to the acquired image information.
10 . The ISP of claim 1 , further comprising a module configured to delete a minimum pixel value from respective image planes acquired from the CFA image sensor before passing corrected image plane information to the demosaicing module.
11 . The ISP of claim 1 , further comprising a geometry correction module configured to compensate for geometry distortions in the image channels acquired from the CFA image sensor before passing corrected image information to the demosaicing module.
12 . The ISP of claim 1 , further configured to automatically focus, expose and or adjust white balance of an image successively acquired by the image sensor based on statistics extracted from a previously acquired image.
13 . The ISP of claim 1 , further comprising a module configured to correct gain and color of demosaiced image plane information before outputting a final image.
14 . The ISP of claim 1 , further comprising a module configured to perform dynamic range compression of demosaiced image plane information before outputting a final image.
15 . An image acquisition system comprising an image sensor and the ISP of claim 1 .