Global approximation to spatially varying tone mapping operators
Techniques to generate global tone-mapping operators (G-TMOs) that, when applied to high dynamic range images, visually approximate the use of spatially varying tone-mapping operators (SV-TMOs) are described. The disclosed G-TMOs provide substantially the same visual benefits as SV-TMOs but do not suffer from spatial artifacts such as halos and are, in addition, computationally efficient compared to SV-TMOs. In general, G-TMOs may be identified based on application of a SV-TMO to a down-sampled version of a full-resolution input image (e.g., a thumbnail). An optimized mapping between the SV-TMO's input and output constitutes the G-TMO. It has been unexpectedly discovered that when optimized (e.g., to minimize the error between the SV-TMO's input and output), G-TMOs so generated provide an excellent visual approximation to the SV-TMO (as applied to the full-resolution image).
1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
receive a high dynamic range color input image;
obtain a grayscale version of the high dynamic range color input image;
downsample the grayscale version of the high dynamic range color input image to generate a down-sampled grayscale input image;
apply a spatially varying tone mapping operator (SV-TMO) to the down-sampled grayscale input image to generate a down-sampled grayscale output image;
determine a global tone mapping operator (G-TMO) based, at least in part, on the down-sampled grayscale input image and the down-sampled grayscale output image; and
apply the G-TMO to the grayscale version of the high dynamic range color input image to generate a low dynamic range grayscale output image;
wherein the instructions to determine the G-TMO further comprise instructions to cause the one or more processors to:
determine a first G-TMO in accordance with a Pool-Adjacent-Violators-Algorithm; and
apply a smoothing filter to the first G-TMO to generate the G-TMO.
2. The non-transitory program storage device of claim 1 , further comprising instructions to cause the one or more processors to convert the low dynamic range grayscale output image to a low dynamic range color output image.
3. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to obtain a grayscale version of the high dynamic range color input image comprise instructions to cause the one or more processors to obtain a brightness channel of the high dynamic range color input image, wherein the high dynamic range color input image comprises a brightness channel and one or more chrominance channels.
4. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to generate a down-sampled grayscale input image comprise instructions to cause the one or more processors to generate a thumbnail of the grayscale version of the high dynamic range color input image.
5. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to determine a global tone mapping operator (G-TMO) comprise instructions to cause the one or more processors to determine a mapping between each pixel in the down-sampled grayscale input image to a corresponding pixel in the down-sampled grayscale output image, wherein the mapping is selected to minimize a specified error criterion.
6. The non-transitory program storage device of claim 5 , wherein the specified error criterion comprises a root means squares error criterion.
7. The non-transitory program storage device of claim 1 , wherein the instructions to cause the one or more processors to determine the first G-TMO in accordance with a Pool-Adjacent-Violators-Algorithm comprise instructions to cause the one or more processors to quantize values from the down-sampled grayscale input image into a specified number of levels, wherein the specified number of levels is less than the number of possible brightness levels that a pixel in the down-sampled grayscale input image may have.
8. The non-transitory program storage device of claim 1 wherein the instructions to cause the one or more processors to generate a grayscale output image comprise instructions to cause the one or more processors to:
apply the G-TMO to the grayscale version of the high dynamic range color input image to generate a first grayscale image; and
perform detail recovery operations on the first grayscale image to generate a grayscale output image.
9. The non-transitory program storage device of claim 8 , wherein the instructions to cause the one or more processors to perform detail recovery operations comprise instructions to cause the one or more processors to apply a bilateral filter to a combination of the grayscale version of the high dynamic range color input image and the first grayscale image.
10. The non-transitory program storage device of claim 8 wherein the instructions to cause the one or more processors to generate a grayscale output image comprise instructions to cause the one or more processors to:
perform detail recovery operations on the first grayscale image to generate a second grayscale image; and
apply an unsharp mask to the second grayscale image to generate a grayscale output image.
11. The non-transitory program storage device of claim 1 , wherein each pixel in the color output image is based, in part, on a ratio of the corresponding pixels in the grayscale output image and the grayscale version of the color input image.
12. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a full-resolution high dynamic range color input image (IN);
obtain a full-resolution grayscale version of image IN (G);
obtain a down-sampled version of image G (T);
apply image T to a spatially varying tone mapping operator (SV-TMO) to generate image T′;
determine a global tone mapping operator (G-TMO) based, at least in part, on images T and T′, wherein the G-TMO approximates operation of the SV-TMO;
apply the G-TMO to image G to generate image G′;
perform detail recovery operations on image G′ to generate image D, wherein performing the detail recovery operations is based, at least in part, on application of a bilateral filter;
determine a full-resolution low dynamic range color output image (O) based, at least in part, on images IN, G and D; and
store image O in a memory.
13. The non-transitory program storage device of claim 12 , wherein the instructions to cause the one or more processors to obtain a down-sampled version of image G comprise instructions to cause the one or more processors to obtain a thumbnail representation of image G.
14. The non-transitory program storage device of claim 13 , wherein the instructions to cause the one or more processors to determine a global tone mapping operator comprise instructions to cause the one or more processors to determine a mapping between each pixel value in image T and a corresponding pixel value in T′, wherein the mapping comprises a function {circumflex over (m)}( ) that minimizes
∑
i
=
1
n
(
T
i
′
-
m
^
(
T
i
)
)
2
,
where n represents the number of pixel values in images T and T′, T i represents the i-th pixel value in image T, and T′ i represents the i-th pixel value in image T′.
15. The non-transitory program storage device of claim 14 , wherein the function {circumflex over (m)}( ) that minimizes
∑
i
=
1
n
(
T
i
′
-
m
^
(
T
i
)
)
2
,
is determined in accordance with a Pool-Adjacent-Value-Algorithm.
16. The non-transitory program storage device of claim 15 , further comprising the application of a smoothing filter to the mapping function {circumflex over (m)}( ).
17. The non-transitory program storage device of claim 12 , further comprising instructions to cause the one or more processors to apply an unsharp mask to image D to generate image D′.
18. The non-transitory program storage device of claim 17 , wherein the instructions to cause the one or more processors to determine a full-resolution low dynamic range color output image (O) comprise instructions to cause the one or more processors to perform a color reconstruction operation based, at least in part, on images IN, G and D′.
19. An electronic device, comprising:
a display element;
a memory operatively coupled to the display element; and
one or more processing units operatively coupled to the display element and the memory, and adapted to execute instructions stored in the memory to:
receive a high dynamic range color input image,
obtain a grayscale version of the color input image,
downsample the grayscale version of the color input image to generate a down-sampled grayscale input image,
apply a spatially varying tone mapping operator (SV-TMO) to the down-sampled grayscale input image to generate a down-sampled grayscale output image,
determine a global tone mapping operator (G-TMO) based;
determining a first G-TMO in accordance with a Pool-Adjacent-Violators-Algorithm; and
applying a smoothing filter to the first G-TMO to generate the G-TMO;
apply the G-TMO to the grayscale version of the color input image to generate a grayscale output image, and
convert the grayscale output image to a low dynamic range color output image.
20. An image conversion method, comprising:
receiving a high dynamic range color input image;
obtaining a grayscale version of the high dynamic range color input image;
down-sampling the grayscale version of the high dynamic range color input image to generate a down-sampled grayscale input image;
applying a spatially varying tone mapping operator (SV-TMO) to the down-sampled grayscale input image to generate a down-sampled grayscale output image;
determining a global tone mapping operator (G-TMO) based on:
determining a first G-TMO in accordance with a Pool-Adjacent-Violators-Algorithm; and
applying a smoothing filter to the first G-TMO to generate the G-TMO; and
applying the G-TMO to the grayscale version of the high dynamic range color input image to generate a low dynamic range grayscale output image.
21. An electronic device, comprising:
a display element;
a memory operatively coupled to the display element; and
one or more processing units operatively coupled to the display element and the memory, and adapted to execute instructions stored in the memory to:
obtain a full-resolution high dynamic range color input image (IN);
obtain a full-resolution grayscale version of image IN (G);
obtain a down-sampled version of image G (T);
apply image T to a spatially varying tone mapping operator (SV-TMO) to generate image T′;
determine a global tone mapping operator (G-TMO) based, at least in part, on images T and T′, wherein the G-TMO approximates operation of the SV-TMO;
apply the G-TMO to image G to generate image G′;
perform detail recovery operations on image G′ based, at least in part, on application of a bilateral filter to generate image D;
determine a full-resolution low dynamic range color output image (O) based, at least in part, on images IN, G and D; and
store image O in a memory.
22. The electronic device of claim 21 , wherein the instructions to cause the one or more processing units to obtain a down-sampled version of image G comprise instructions to cause the one or more processors to obtain a thumbnail representation of image G.
23. The electronic device of claim 22 , wherein the instructions to cause the one or more processing units to determine a global tone mapping operator comprise instructions to cause the one or more processors to determine a mapping between each pixel value in image T and a corresponding pixel value in T′, wherein the mapping comprises a function {circumflex over (m)}( ) that minimizes
∑
i
=
1
n
(
T
i
′
-
m
^
(
T
i
)
)
2
,
where n represents the number of pixel values in images T and T′, T i represents the i-th pixel value in image T, and T′ i represents the i-th pixel value in image T′.
24. The electronic device of claim 23 , wherein the instructions to determine the function {circumflex over (m)}( ) that minimizes
∑
i
=
1
n
(
T
i
′
-
m
^
(
T
i
)
)
2
,
further comprise instructions to cause the one or more processing units to determine the function {circumflex over (m)}( ) in accordance with a Pool-Adjacent-Value-Algorithm.
25. The electronic device of claim 24 , wherein the instructions further comprise instructions to cause the one or more processing units to apply a smoothing filter to the mapping function {circumflex over (m)}( ).
26. The electronic device of claim 21 , wherein the instructions further comprise instructions to cause the one or more processing units to cause the one or more processors to apply an unsharp mask to image D to generate image D′.
27. The electronic device of claim 26 , wherein the instructions to cause the one or more processors to determine a full-resolution low dynamic range color output image (O) further comprise instructions to cause the one or more processing units to cause the one or more processors to perform a color reconstruction operation based, at least in part, on images IN, G and D′.