IP Library Granted Patent US 10,530,995
Granted Patent B2
US 10,530,995 · App. 15/821,032 · Granted Jan 7, 2020

Global tone mapping

Inventors: Bruno César Douady-Pleven (Bures-sur-Yvette, FR); Michael Serge Andre Kraak (Cachan, FR); Guillaume Matthieu Guérin (Paris, FR); Thomas Nicolas Emmanuel Veit (Meudon, FR)
Assignee: GoPro, Inc.
H04N5/23229G06T3/4015G06T5/004G06T5/009G06T5/50H04N1/4074H04N1/603H04N1/6008H04N5/232H04N5/2355H04N5/357H04N5/3572H04N9/045H04N9/643H04N9/646H04N9/68H04N9/735H04N9/77G06T2207/10016G06T2207/10024G06T2207/20012G06T2207/20182H04N2209/046
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,530,995
App. No.
15/821,032
Granted
Jan 7, 2020
Kind
B2
Abstract

A system accesses an image with each pixel of the image having luminance values each representative of a color component of the pixel. The system generates a first histogram for aggregate luminance values of the image, and accesses a target histogram for the image representative of a desired global image contrast. The system computes a transfer function based on the first histogram and the target histogram such that when the transfer function is applied, a histogram of the modified aggregate luminance values is within a threshold similarity of the target histogram. The system modifies the image by applying the transfer function to the luminance values of the image to produce a tone mapped image, and outputs the modified image.

Claims (99)

1. A system comprising:

an image sensor;

a memory; and

a processor, wherein the memory stores instructions executable by the processor to cause the system to:

access an image captured by the image sensor;

determine a first histogram for luminance values of the image;

access a target histogram for the image;

compute a transfer function based on the first histogram and the target histogram such that when the transfer function is applied to pixel values of the image to create a set of modified luminance values, a histogram of the modified luminance values is within a threshold similarity of the target histogram;

generate an exposure correction curve based on a lookup table, the exposure correction curve compensating for portions of the first histogram that are representative of an underexposure of the image within a corresponding portion of a color channel, and the lookup table producing a linear output for brightness values below a threshold and a compressed non-linear output for brightness values above a threshold;

apply the exposure correction curve to pixel values of the image to generate exposure corrected values of the image;

transpose the transfer function from a post-gamma curve space to a linear space of the image;

apply the transfer function to pixel values of the image to produce a tone mapped image; and

output the tone mapped image.

2. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

compute, for each pixel of the image, a luminance value based on a luminance function applied to a set of values corresponding to respective color components of the pixel.

3. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

generate a histogram for a color channel of the image; and

apply a blackpoint to the image for the color channel based on the generated histogram for the color channel to establish a lower bound for the color channel.

4. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

adjust the transfer function to remove above-threshold amplification of noise resulting from an application of the transfer function to the image.

5. The system of claim 4 , wherein the memory stores instructions executable by the processor to cause the system to:

compute a derivative of the transfer function;

adjust one of a high threshold area and a low threshold area of the derivative of the transfer function such that the high threshold area and the low threshold area are equal;

modify the derivative by removing the high threshold area and the low threshold area from the derivative; and

compute a modified transfer function based on the modified derivative.

6. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

apply an unsharp mask to portions of the image that include areas of interest in order to increase local contrast.

7. The system of claim 6 , wherein the memory stores instructions executable by the processor to cause the system to:

determine one or more changed image portions of the tone mapped image that are one of compressed and amplified compared to a corresponding image portion of the image; and

modify these changed image portions to increase the local contrast in these changed image portions.

8. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

in response to a detecting that the image is uniform, modify the transfer function to reduce a gain of the transfer function.

9. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

obtain scene classification information for the image; and

modify the transfer function based on the scene classification information.

10. The system of claim 9 , wherein the scene classification information is determined by an automatic white balance module configured to apply white balance correction to the image.

11. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

determine a blue sky indication that reflects a proportion of a scene depicted in the image that includes blue sky; and

modify the transfer function based on the blue sky indication to reduce a gain of the transfer function.

12. The system of claim 11 , wherein the blue sky indication is determined by an automatic white balance module configured to apply white balance correction to the image.

13. The system of claim 1 , wherein the memory stores instructions executable by the processor to cause the system to:

determine a faces indication that reflects a proportion of a scene depicted in the image that includes one or more human faces; and

modify the transfer function based on the faces indication to reduce a gain of the transfer function.

14. A method comprising:

accessing an image captured by an image sensor;

computing a first histogram for luminance values of the image;

accessing a target histogram for the image;

computing a transfer function based on the first histogram and the target histogram such that when the transfer function is applied to pixel values of the image to create a set of modified luminance values, a histogram of the modified luminance values is within a threshold similarity of the target histogram;

obtaining scene classification information for the image;

modifying the transfer function based on the scene classification information, wherein modifying the transfer function comprises multiplying a difference of a gain of the transfer function and one by an adjustment parameter that is modulated as a function of the scene classification information;

applying the transfer function to pixel values of the image to produce a tone mapped image; and

outputting the tone mapped image.

15. The method of claim 14 , wherein the scene classification information is determined by an automatic white balance module configured to apply white balance correction to the image.

16. The method of claim 14 , comprising:

determining a blue sky indication that reflects a proportion of a scene depicted in the image that includes blue sky; and

modifying the transfer function based on the blue sky indication to reduce a gain of the transfer function.

17. The method of claim 16 , wherein the blue sky indication is determined by an automatic white balance module configured to apply white balance correction to the image.

18. The method of claim 14 , comprising:

determining a faces indication that reflects a proportion of a scene depicted in the image that includes one or more human faces; and

modifying the transfer function based on the faces indication to reduce a gain of the transfer function.

19. The method of claim 14 , comprising:

transposing the transfer function from a post-gamma curve space to a linear space.

20. The method of claim 14 , comprising:

adjusting the transfer function to remove above-threshold amplification of noise resulting from an application of the transfer function to the image.

21. The method of claim 14 , comprising:

computing a derivative of the transfer function;

adjusting one of a high threshold area and a low threshold area of the derivative of the transfer function such that the high threshold area and the low threshold area are equal;

modifying the derivative by removing the high threshold area and the low threshold area from the derivative; and

computing a modified transfer function based on the modified derivative.

22. The method of claim 14 , comprising:

applying an unsharp mask to portions of the image that include areas of interest in order to increase local contrast.

23. The method of claim 14 , comprising:

computing a gain in a linear domain equivalent to the transfer function previously estimated in a non-linear domain.

24. The method of claim 14 , comprising:

in response to a detecting that the image is uniform, modifying the transfer function to reduce a gain of the transfer function.

25. The method of claim 14 , comprising:

generating an exposure correction curve based on a lookup table, the exposure correction curve compensating for portions of the first histogram of the values that are representative of an underexposure of the image within a corresponding portion of a color channel, the lookup table producing a linear output for brightness values below a threshold and a compressed non-linear output for brightness values above a threshold;

applying the exposure correction curve to pixel values of the image to generate exposure corrected values of the image; and

transposing the transfer function from a post-gamma curve space to a linear space of the image.

26. The method of claim 14 , comprising:

generating a histogram for a color channel of the image; and

applying a blackpoint to the image for the color channel based on the generated histogram for the color channel to establish a lower bound for the color channel.

27. An apparatus comprising:

an image sensor configured to capture an image;

a histogram module configured to compute a first luminance histogram of the image based on luminance values of the image; and

a histogram transfer module configured to modify pixel values of the image by applying a transfer function to produce a tone mapped image such that a second luminance histogram for the tone mapped image matches a target histogram, wherein the histogram transfer module is configured to:

compute a derivative of the transfer function;

adjust one of a high threshold area and a low threshold area of the derivative of the transfer function such that the high threshold area and the low threshold area are equal;

modify the derivative of the transfer function by removing the high threshold area and the low threshold area from the derivative; and

compute a modified transfer function from the modified derivative of the transfer function.

28. The apparatus of claim 27 , comprising:

a target histogram determination module configured to determine the target histogram.

29. The apparatus of claim 28 , wherein the target histogram determination module is configured to:

select the target histogram from among a set of histograms based on input received from a user; and

present to the user an estimated preview of how a histogram from the set of histograms would affect the image.

30. The apparatus of claim 27 , comprising:

an automatic white balance module configured to determine a blue sky indication that reflects a proportion of a scene depicted in the image that includes blue sky and apply white balance correction, based on the blue sky indication, to the image, and wherein the histogram transfer module is configured to modify the transfer function based on the blue sky indication to reduce a gain of the transfer function.

31. The apparatus of claim 27 , comprising:

an unsharp mask module configured to segment the image based on regions that are compressed by a gain from the transfer function and the regions that are amplified by a gain from the transfer function.

Assignments (7)
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: WELLS FARGO BANK, NATIONAL ASSOCIATION, AS AGENT
Reel/Frame 072358/0001 →
SECURITY INTEREST Recorded Aug 4, 2025
From: GOPRO, INC.
To: FARALLON CAPITAL MANAGEMENT, L.L.C., AS AGENT
Reel/Frame 072340/0676 →
RELEASE OF PATENT SECURITY INTEREST Recorded Jan 25, 2021
From: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
To: GOPRO, INC.
Reel/Frame 055106/0434 →
SECURITY INTEREST Recorded Oct 19, 2020
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 054113/0594 →
SECURITY INTEREST Recorded Jun 7, 2018
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 047140/0831 →
SECURITY INTEREST Recorded Feb 9, 2018
From: GOPRO, INC.
To: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT
Reel/Frame 044983/0718 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 29, 2017
From: DOUADY-PLEVEN, BRUNO CÉSAR; KRAAK, MICHAEL SERGE ANDRE; GUÉRIN, GUILLAUME MATTHIEU; VEIT, THOMAS NICOLAS EMMANUEL
To: GOPRO, INC.
Reel/Frame 044246/0897 →
Continuity (5)
Continuation In Part 15081682 · Mar 25, 2016
Continuation In Part 15081688 · Mar 25, 2016
Provisional Application 62268687 · Dec 17, 2015
Provisional Application 62174786 · Jun 12, 2015
Related Publication 20180097992A1 · Apr 5, 2018
Cited By (1)
US 12,206,998