High-resolution image capture by luminance-driven upsampling of pixel-binned image sensor array output
Techniques are described for efficient high-resolution output of an image captured using a high-pixel-count image sensor based on pixel binning followed by luminance-guided umsampling. For example, an image sensor array is configured according to a red-green-blue-luminance (RGBL) CFA pattern, such that at least 50-percent of the imaging pixels of the array are luminance (L) pixels. Pixel binning is used during readout of the array to concurrently generate a downsampled RGB capture frame and a downsampled L capture frame. Following the readout, the L capture frame is upsampled (e.g., by upscaling and interpolation) to generate an L guide frame with 100-percent luminance density. An upsampled RGB frame can then be generated by interpolating the RGB capture frame based both on known neighboring RGB information (e.g., from the RGB capture frame and previously interpolated information), as adjusted based on local luminance information from the L guide frame.
1. A method for high-resolution image capture using an image sensor system, the method comprising:
obtaining a plurality of photo-sensor output levels, each corresponding to an amount of photocharge accumulated by a respective one of a plurality of photo-sensors of an image sensor array responsive to exposing the image sensor array to incident illumination during an integration window, each of the plurality of photo-sensors optically coupled with a respective color filter of a color filter array (CFA) in accordance with a red-green-blue-luminance (RGBL) CFA pattern;
reading out the plurality of photo-sensor output levels using pixel binning, in a readout window subsequent to the integration window, to generate a downsampled RGB capture frame and a downsampled L capture frame; and
generating a luminance-enhanced output image, in an upsampling time window following the readout time window, by upsampling the L capture frame to generate an L guide frame, and applying luminance-guided interpolation to the RGB capture frame based on the L guide frame to upsample the RGB capture frame.
2. The method of claim 1 , further comprising:
outputting the luminance enhanced output image to a processor-controlled system separate from the image sensor system.
3. The method of claim 1 , wherein the upsampling the L capture frame to generate the L guide frame comprises:
upscaling the L capture frame to generate an upscaled L frame having a target frame size defined by an output image schema of the luminance-enhanced output image, a portion of the frame populated based on known luminance values from the L capture frame; and
upsampling the upscaled L frame to generate the L guide frame based on interpolating the known luminance values, such that the L guide frame has 100-percent density of luminance information.
4. The method of claim 1 , wherein the applying luminance-guided interpolation to the RGB capture frame comprises, for each color plane of the RGB capture frame:
upscaling the color plane to generate a respective upscaled color frame, such that the respective upscaled color frame has a target frame size defined by an output image schema of the luminance-enhanced output image, and a portion of the respective upscaled color frame is populated based on respective known color values from the color plane; and
upsampling the respective upscaled color frame to generate a respective upsampled color frame by applying the luminance-guided interpolation to the respective known color values based on the L guide frame.
5. The method of claim 4 , wherein
the output image schema further defines a target CFA pattern for the luminance-enhanced output image; and
for each color plane of the RGB capture frame, the upsampling the respective upscaled color frame generates the respective upsampled color frame by using the luminance-guided interpolation to remosaic the respective known color values based on the L guide frame into the target CFA pattern.
6. The method of claim 1 , wherein:
the L guide frame comprises an array of L value cells, each having a respective luminance value; and
the applying luminance-guided interpolation to the RGB capture frame comprises generating an upsampled RGB frame to have an array of color value cells by, for each color value cell of a portion of the color value cells:
generating a respective initial value for the color value cell based on interpolating a respective neighboring set of known color values based on the RGB capture frame; and
computing an estimated color pixel value for the color value cell by adjusting the respective initial value based on the luminance values of those of the L value cells spatially corresponding to the color value cell and to the locations of the respective neighboring set of known color values.
7. The method of claim 6 , wherein the adjusting comprises:
computing a color-ratio or a color-difference between the L value cell spatially corresponding to the color value cell, and those of the L value cells spatially corresponding to the locations of the respective neighboring set of known color values; and
adjusting the initial value of the color value cell based on the color-ratio or the color-difference.
8. The method of claim 1 , wherein:
the image sensor array has a native resolution;
the RGB capture frame has a downsampled resolution that is less than or equal to one-fourth of the native resolution; and
the luminance-enhanced output image has an output resolution that is at least three times the downsampled resolution.
9. The method of claim 8 , wherein the output resolution is equal to the native resolution.
10. The method of claim 1 , wherein the RGBL CFA pattern of the image sensor array has a 50-percent density of luminance imaging pixels and a 50-percent density of color imaging pixels.
11. The method of claim 1 , wherein the RGBL CFA pattern of the image sensor array has a 75-percent density of luminance imaging pixels and a 25-percent density of color imaging pixels.
12. The method of claim 1 , wherein the pixel binning is charge binning.
13. An image sensor comprising:
a photo-sensor array comprising a plurality of photo-sensor, each photo-sensor optically coupled with a respective color filter of a color filter array (CFA) in accordance with a red-green-blue-luminance (RGBL) CFA pattern, the plurality of photo-sensors configured to generate a plurality of photo-sensor output levels corresponding to an amount of photocharge accumulated responsive to exposure to incident illumination during an integration window;
binning readout circuitry coupled with the photo-sensor array to read out the plurality of photo-sensor output levels using pixel binning, in a readout window subsequent to the integration window, to generate a downsampled RGB capture frame and a downsampled L capture frame; and
an upsampling engine coupled with the binning readout circuitry to generate a luminance-enhanced output image, in an upsampling time window following the readout time window, by upsampling the L capture frame to generate an L guide frame, and applying luminance-guided interpolation to the RGB capture frame based on the L guide frame to upsample the RGB capture frame.
14. The image sensor of claim 13 , further comprising:
a sensor controller coupled with at least the binning readout circuitry and the upsampling engine, such that:
the sensor controller is to generate and output first control signals to direct the binning readout circuitry to read out the plurality of photo-sensor output levels using the pixel binning; and
the sensor controller is to generate and output second control signals to direct the upsampling engine to generate the luminance-enhanced output image.
15. The image sensor of claim 13 , wherein the upsampling engine is to upsample the L capture frame to generate the L guide frame by:
upscaling the L capture frame to generate an upscaled L frame having a target frame size defined by an output image schema of the luminance-enhanced output image, a portion of the frame populated based on known luminance values from the L capture frame; and
upsampling the upscaled L frame to generate the L guide frame based on interpolating the known luminance values, such that the L guide frame has 100-percent density of luminance information.
16. The image sensor of claim 13 , wherein the upsampling engine is to apply the luminance-guided interpolation to the RGB capture frame, for each color plane of the RGB capture frame, by:
upscaling the color plane to generate a respective upscaled color frame, such that the respective upscaled color frame has a target frame size defined by an output image schema of the luminance-enhanced output image, and a portion of the respective upscaled color frame is populated based on respective known color values from the color plane; and
upsampling the respective upscaled color frame to generate a respective upsampled color frame by applying the luminance-guided interpolation to the respective known color values based on the L guide frame.
17. The image sensor of claim 16 , wherein
the output image schema further defines a target CFA pattern for the luminance-enhanced output image; and
for each color plane of the RGB capture frame, the upsampling the respective upscaled color frame generates the respective upsampled color frame by using the luminance-guided interpolation to remosaic the respective known color values based on the L guide frame into the target CFA pattern.
18. The image sensor of claim 13 , wherein:
the L guide frame comprises an array of L value cells, each having a respective luminance value; and
the upsampling engine is to apply the luminance-guided interpolation to the RGB capture frame by generating an upsampled RGB frame to have an array of color value cells by, for each color value cell of a portion of the color value cells:
generating a respective initial value for the color value cell based on interpolating a respective neighboring set of known color values based on the RGB capture frame; and
computing an estimated color pixel value for the color value cell by adjusting the respective initial value based on the luminance values of those of the L value cells spatially corresponding to the color value cell and to the locations of the respective neighboring set of known color values.
19. The image sensor of claim 13 , wherein:
the image sensor array has a native resolution;
the RGB capture frame has a downsampled resolution that is less than or equal to one-fourth of the native resolution; and
the luminance-enhanced output image has an output resolution that is at least three times the downsampled resolution.
20. The image sensor of claim 13 , wherein the RGBL CFA pattern of the image sensor array has at least a 50-percent density of luminance imaging pixels.