Granted Patent B2

US 12,229,914 · App. 17/616,544 · Granted Feb 18, 2025

Inventor: Jun Nishimura (San Jose, CA)

Assignee: Google LLC

G06T3/4015G06T1/20G06T7/90G06T11/001G06T2207/10024

Abstract

Claims (54)

1. A system comprising:

an image capture device configured to generate raw image data of an image, the raw image data comprising, for each of a plurality of highlight regions in the image, original channel values for one or more channels of the highlight region; and

an image signal processor comprising:

one or more components of an image processing pipeline that are configured to process the raw image data to generate updated image data, wherein the updated image data comprises updated channel values for each highlight region in the image, and wherein, for each of one or more highlight regions of the image, one or more updated channel values of the highlight region exceed a maximum channel value according to a predetermined data precision, and

a highlight recovery circuit configured to perform a hue correction process on the updated channel values of the one or more highlight regions of the image, wherein performing the hue correction process comprises processing the updated channel values to generate final channel values that i) satisfy the predetermined data precision and ii) recover hue values of the one or more highlight regions of the image, wherein performing the hue correction process comprises, for each highlight region in the image:

calculating hue values including adjusting the respective updated channel values for each of the one or more channels of the highlight region, wherein the hue values represent a proportion of the updated channel values of the channels;

generating a clipped channel value for each of the one or more channels, wherein the clipped channel value equals the respective updated channel value if the updated channel value is less than or equal to the maximum channel value, and equals the maximum channel value if the updated channel value is greater than the maximum channel value; and

calculating a final channel value for each of the one or more channels including adjusting the respective clipped channel values using the respective hue values to emulate the proportion of the updated channel values of the channels.

2. The system of claim 1 , wherein the highlight regions are individual pixels and wherein the one or more channels comprise an R channel, a G channel, and a B channel.

3. The system of claim 1 , wherein the highlight regions are groups of four pixels arranged in two-by-two grids in a color filter array, wherein each of the pixels has a single value representing a single color, and wherein the one or more channels comprise the values of the four pixels.

4. The system of claim 1 , wherein calculating, for each highlight region in the image, hue values further comprises:

determining a measure of central tendency of the respective updated channel values of the one or more channels of the highlight region; and

calculating hue values including adjusting the respective updated channel values for each of the one or more channels using the measure of central tendency of the updated channel values of the highlight region.

5. The system of claim 4 , wherein calculating, for each highlight region in the image, a final channel value for each of the one or more channels comprises:

determining a measure of the central tendency of the clipped channel values of the one or more channels; and

calculating a final channel value for each of the one or more channels including adjusting the respective clipped channel values using the respective hue values and the measure of central tendency of the clipped channel values of the highlight region.

6. The system of claim 5 , wherein for each highlight region:

the measure of central tendency of the updated channel values and the clipped channel values of the one or more channels is a mean of the respective channel values; and

calculating the hue values of the one or more channels comprises:

generating initial hue values by calculating the difference between the updated channel values and the mean of the updated channel values; and

generating the hue values by normalizing the initial hue values comprising dividing the initial hue values by a length of a first vector wherein each element of the first vector is an initial hue value.

7. The system of claim 6 , wherein calculating the final channel values of the one or more channels comprises:

generating scaled hue values by scaling the hue values by a length of a second vector, wherein each element of the second vector is a difference between the clipped channel value of a channel and the mean of the clipped channel values; and

generating the final channel values by adding the scaled hue values to the mean of the clipped channel values.

8. A method comprising:

receiving raw image data of an image, the raw image data comprising, for each of a plurality of highlight regions in the image, original channel values for one or more channels of the highlight region;

processing the raw image data to generate updated image data, wherein the updated image data comprises updated channel values for each highlight region in the image, and wherein, for each of one or more highlight regions of the image, one or more updated channel values of the highlight region exceed a maximum channel value according to a predetermined data precision; and

performing a hue correction process on the updated channel values of the one or more highlight regions of the image, wherein performing the hue correction process comprises processing the updated channel values to generate final channel values that i) satisfy the predetermined data precision and ii) recover hue values of the one or more highlight regions of the image, wherein performing the hue correction process comprises, for each highlight region in the image:

calculating hue values including adjusting the respective updated channel values for each of the one or more channels of the highlight region, wherein the hue values represent a proportion of the updated channel values of the channels;

generating a clipped channel value for each of the one or more channels, wherein the clipped channel value equals the respective updated channel value if the updated channel value is less than or equal to the maximum channel value, and equals the maximum channel value if the updated channel value is greater than the maximum channel value; and

calculating a final channel value for each of the one or more channels including adjusting the respective clipped channel values using the respective hue values to emulate the proportion of the updated channel values of the channels.

9. The method of claim 8 , wherein the highlight regions are individual pixels and wherein the one or more channels comprise an R channel, a G channel, and a B channel.

10. The method of claim 8 , wherein the highlight regions are groups of four pixels arranged in two-by-two grids in a color filter array, wherein each of the pixels has a single value representing a single color, and wherein the one or more channels comprise the values of the four pixels.

11. The method of claim 8 , wherein calculating, for each highlight region in the image, hue values further comprises:

determining a measure of central tendency of the updated channel values of the one or more channels of the highlight region; and

calculating hue values including adjusting the updated channel values for each of the one or more channels using the measure of central tendency of the updated channel values of the highlight region.

12. The method of claim 11 , wherein calculating, for each highlight region in the image, a final channel value for each of the one or more channels comprises:

determining a measure of the central tendency of the clipped channel values of the one or more channels; and

calculating a final channel value for each of the one or more channels including adjusting the respective clipped channel values using the respective hue values and the measure of central tendency of the clipped channel values.

13. The method of claim 12 , wherein for each highlight region:

the measure of central tendency of the updated channel values and the clipped channel values of the one or more channels is a mean of the respective channel values; and

calculating the hue values of the one or more channels comprises:

generating initial hue values by calculating the difference between the updated channel values and the mean of the updated channel values; and

generating the hue values by normalizing the initial hue values comprising dividing the initial hue values by a length of a first vector wherein each element of the first vector is an initial hue value.

14. The method of claim 13 , wherein calculating the final channel values of the one or more channels comprises:

generating scaled hue values by scaling the hue values by a length of a second vector, wherein each element of the second vector is a difference between the clipped channel value of a channel and the mean of the clipped channel values; and

generating the final channel values by adding the scaled hue values to the mean of the clipped channel values.

15. One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations, the operations comprising:

receiving raw image data of an image, the raw image data comprising, for each of a plurality of highlight regions in the image, original channel values for one or more channels of the highlight region;

processing the raw image data to generate updated image data, wherein the updated image data comprises updated channel values for each highlight region in the image, and wherein, for each of one or more highlight regions of the image, one or more updated channel values of the highlight region exceed a maximum channel value according to a predetermined data precision; and

performing a hue correction process on the updated channel values of the one or more highlight regions of the image, wherein performing the hue correction process comprises processing the updated channel values to generate final channel values that i) satisfy the predetermined data precision and ii) recover hue values of the one or more highlight regions of the image, wherein performing the hue correction process comprises, for each highlight region in the image:

calculating hue values including adjusting the respective updated channel values for each of the one or more channels of the highlight region, wherein the hue values represent a proportion of the updated channel values of the channels;

generating a clipped channel value for each of the one or more channels, wherein the clipped channel value equals the respective updated channel value if the updated channel value is less than or equal to the maximum channel value, and equals the maximum channel value if the updated channel value is greater than the maximum channel value; and

calculating a final channel value for each of the one or more channels including adjusting the respective clipped channel values using the respective hue values to emulate the proportion of the updated channel values of the channels.

Assignments (1)

Continuity (2)

References Cited (34)