Highlight recovery for image sensors
Techniques are generally described for highlight recovery for image sensors. In some examples, a first frame of image data comprising a first pixel may be received. The first pixel may include a value C i,c where c∈(R, G, B, IR). A white balance gained value C i,c wb may be determined using C i,c ·WB c , where WB c represents a per-color channel white balance gain. A color-corrected value C i ccm may be determined based at least in part on C i,c wb and a color correction matrix. An interpolation coefficient α may be determined based at least in part on a clipped representation of the color-corrected value C i ccm . A second value C i out may be determined for the first pixel by interpolating between an unclipped representation of the first pixel and an unclipped representation of C i ccm using α.
1. A method comprising:
receiving a first frame of image data representing a plurality of pixels arranged in a grid, wherein a first pixel of the plurality of pixels is associated with an input red pixel value C R , an input green pixel value C G , an input blue pixel value C B , and an input infrared pixel value C IR ;
determining a white-balanced red pixel value C R wb for the first pixel by multiplying the input red pixel value C R by a white balance gain corresponding to a red channel WB R ;
determining a clipped, white-balanced red pixel value C R wb,clipped by replacing white-balanced pixel values of C R wb greater than 1 with a value of 1 and replacing white-balanced pixel values of C R wb less than 0 with a value of 0;
determining a color-corrected red pixel value C R ccm , a color-corrected green pixel value C G ccm , and a color-corrected blue pixel value C B ccm by multiplying a 1×4 matrix by a 4×3 color correction matrix, wherein the 1×4 matrix comprises the clipped, white-balanced red pixel value C R WB,clipped , a clipped, white-balanced green pixel value C G wb,clipped , a clipped, white-balanced blue pixel value C B wb,clipped , and a clipped, white-balanced infrared pixel value C IR wb,clipped ;
determining a clipped, color-corrected red pixel value C R ccm,clipped by replacing color corrected red pixel values C R ccm greater than 1 with the value of 1 and replacing color-corrected red pixel values C R ccm less than 0 with the value of 0;
determining an interpolation coefficient α for the first pixel based on the clipped, color-corrected red pixel value C R ccm,clipped , a clipped, color-corrected green pixel value C G ccm,clipped , and a clipped, color-corrected blue pixel value C B ccm,clipped ;
determining a first luminance value for the first pixel C luma using a weighted combination of the input red pixel value, the input green pixel value, and the input blue pixel value; and
determining a modified red pixel value by interpolating between C luma and an unclipped, color-corrected red pixel value C R ccm,unclipped using the interpolation coefficient α.
2. The method of claim 1 , further comprising:
determining the unclipped, color-corrected red pixel value by multiplying a 1×4 matrix comprising the white-balanced red pixel value C R wb for the first pixel, a white-balanced green pixel value C G wb , a white-balanced blue pixel value C B wb , and a white-balanced infrared pixel value C IR wb by the 4×3 color correction matrix.
3. The method of claim 1 , wherein the interpolation coefficient α increases as an average saturation level of C R ccm,clipped +C G ccm,clipped +C B ccm,clipped increases.
4. The method of claim 3 , further comprising determining the modified red pixel value using:
modified red pixel value=(1−α)·C R com +α·C luma , where C R ccm is the unclipped, color-corrected red pixel value.
5. A method comprising:
receiving first pixel data;
determining an interpolation coefficient based at least in part on a clipped pixel value determined based on the first pixel data;
determining second pixel data based on fusing together
color-corrected pixel values determined based on the first pixel data, and
non-color-corrected pixel values determined based on the first pixel data;
wherein fusing together the color-corrected pixel values and the non-color-corrected pixel values comprises determining, for each respective channel of a plurality of color channels, a respective blended pixel value for the respective channel based on summing together resultant values from
multiplying one of the color-corrected pixel values corresponding to the respective channel by a first value determined based on the interpolation coefficient, and
multiplying one of the non-color-corrected pixel values corresponding to the respective channel by a second value determined based on the interpolation coefficient; and
wherein the second pixel data comprises the blended pixel values.
6. The method of claim 5 , wherein the first pixel data comprises a value corresponding to a first color channel, a value corresponding to a second color channel, a value corresponding to a third color channel, and a value corresponding to an infrared color channel.
7. The method of claim 6 , wherein the method comprises:
determining, based on the first pixel data, a set of white balance gained pixel values including a value corresponding to a first color channel, a value corresponding to a second color channel, a value corresponding to a third color channel, and a value corresponding to an infrared color channel;
determining, based at least in part on the set of white balance gained pixel values and a set of color correction values, a first set of white balanced gained color corrected pixel values, the determining of the first set of white balance gained color corrected pixel values comprising
determining that a third value is outside of a first range,
based on the determining that the third value is outside of the first range,
determining a fourth value based on a value that is inside of the first range;
wherein the interpolation coefficient is determined based at least in part on values of the set of white balance gained color corrected pixel values.
8. The method of claim 7 , wherein the set of color correction values is stored in a color correction matrix.
9. The method of claim 7 , wherein the method further comprises determining, based at least in part on the set of white balance gained pixel values and the set of color correction values, a second set of white balanced gained color corrected pixel values; and
wherein the second set of white balanced gained color corrected pixel values comprises the color-corrected pixel values.
10. The method of claim 7 , wherein the interpolation coefficient is determined based on use of a recovery strength parameter.
11. The method of claim 7 , wherein the method comprises
determining a fifth value based at least in part on summing together values determined based on the set of white balance gained color corrected pixel values,
determining a sixth value based on a configuration parameter,
determining a seventh value based on the fifth value and the sixth value; and
wherein the interpolation coefficient is determined based at least in part on raising Euler's number to a power corresponding to the seventh value.
12. The method of claim 5 , wherein the first pixel data comprises a value corresponding to a first color channel, a value corresponding to a second color channel, a value corresponding to a third color channel, and a value corresponding to an infrared color channel; and
wherein the second pixel data comprises a value corresponding to the first color channel, a value corresponding to the second color channel, and a value corresponding to the third color channel.
13. The method of claim 5 , wherein the non-color-corrected pixel values are luminance values determined based on the first pixel data.
14. The method of claim 7 , wherein the non-color-corrected pixel values comprise values from the set of white balance gained pixel values.
15. The method of claim 14 , wherein the non-color-corrected pixel values comprise a value corresponding to the first color channel, a value corresponding to the second color channel, and a value corresponding to the third color channel.
16. The method of claim 14 , wherein the second value comprises the interpolation coefficient and the first value is determined based on subtracting the interpolation coefficient from one.
17. The method of claim 14 , wherein the first value comprises the interpolation coefficient and the second value is determined based on subtracting the interpolation coefficient from one.
18. The method of claim 5 , wherein the method further comprises providing the second pixel data as input to a gamma correction process, an RGB to YUV conversion process, or a scaling process.
19. A system comprising:
one or more processors;
non-transitory computer-readable memory;
one or more multiplier circuits; and
the one or more processors effective to:
receive first pixel data;
determine an interpolation coefficient based at least in part on a clipped pixel value determined based on the first pixel data;
determine second pixel data based on fusing together
color-corrected pixel values determined based on the first pixel data, and
non-color-corrected pixel values determined based on the first pixel data;
the one or more multiplier circuits effective to:
determine, for each respective channel of a plurality of color channels, a respective blended pixel value for the respective channel based on summing together resultant values by
multiplying one of the color-corrected pixel values corresponding to the respective channel by a first value determined based on the interpolation coefficient, and
multiplying one of the non-color-corrected pixel values corresponding to the respective channel by a second value determined based on the interpolation coefficient; and
wherein the second pixel data comprises the blended pixel values.
20. The system of claim 19 , wherein the first pixel data comprises a value corresponding to a first color channel, a value corresponding to a second color channel, a value corresponding to a third color channel, and a value corresponding to an infrared color channel.
21. The system of claim 20 , wherein the one or more processors are further effective to:
determine, based on the first pixel data, a set of white balance gained pixel values including a value corresponding to a first color channel, a value corresponding to a second color channel, a value corresponding to a third color channel, and a value corresponding to an infrared color channel;
determine, based at least in part on the set of white balance gained pixel values and a set of color correction values, a first set of white balanced gained color corrected pixel values, wherein, to determine the first set of white balance gained color corrected pixel values, the one or more processors are further effective to:
determine that a third value is outside of a first range,
based on the determination that the third value is outside of the first range,
determine a fourth value based on a value that is inside of the first range;
wherein the interpolation coefficient is determined based at least in part on values of the set of white balance gained color corrected pixel values.
22. The system of claim 21 , wherein the set of color correction values is stored in a color correction matrix.