User interface and graphics composition with high dynamic range video
A method and system for adaptively mixing video components with graphics/UI components, where the video components and graphics/UI components may be of different types, e.g., different dynamic ranges (such as HDR, SDR) and/or color gamut (such as WCG). The mixing may result in a frame optimized for a display device's color space, ambient conditions, viewing distance and angle, etc., while accounting for characteristics of the received data. The methods include receiving video and graphics/UI elements, converting the video to HDR and/or WCG, performing statistical analysis of received data and any additional applicable rendering information, and assembling a video frame with the received components based on the statistical analysis. The assembled video frame may be matched to a color space and displayed. The video data and graphics/UI data may have or be adjusted to have the same white point and/or primaries.
1. A computer-implemented compositing method, comprising:
receiving, by a processor, first data from a first source having a first dynamic range and an associated first electro-optical transfer function (EOTF);
receiving, by the processor, second data, from a second source different than the first source, having a second dynamic range that is lower than the first dynamic range and an associated second electro-optical transfer function;
converting, by the processor, the first data to a converted first data having a third dynamic range and an associated third electro-optical transfer function that matches at least a segment of the second electro-optical transfer function;
assembling, by the processor, a video frame from the converted first data and the second data, and
rendering, by the processor, an output frame based on the assembled video frame.
2. The method of claim 1 , further comprising converting, by the processor, the second data to a fourth dynamic range at least partially overlapping with the third dynamic range.
3. The method of claim 1 , wherein the conversion of the first data retains at least one of a white point and primary chromaticities of the first data.
4. The method of claim 1 , wherein the conversion of the first data includes converting the first data to one of a high dynamic range (HDR) and a wide color gamut (WCG).
5. The method of claim 1 , wherein the converted first data and the second data share at least one primary chromaticity.
6. The method of claim 1 , wherein the EOTFs are represented by a gamma curve.
7. The method of claim 1 , wherein the assembly of the video frame is based on at least one of: (i) characteristics of the first data and the second data, (ii) ambient conditions of a display device, and (iii) commands received at a user interface of the display device.
8. The method of claim 1 , further comprising performing statistical analysis based on additional rendering information relating to a display device and includes a level of light leakage, a level of light reflection, a backlight level, a physical size, a white point, and an average pixel brightness.
9. The method of claim 1 , further comprising matching the assembled frame to a color space of the display device.
10. The method of claim 1 , wherein the rendering of the output frame includes at least one of: storing the output frame in a non-transitory computer-readable medium, outputting the output frame to a network, and displaying the output frame on a display device.
11. The method of claim 1 , wherein the second data has a EOTF matching at least a segment of a EOTF of the first dynamic range.
12. A video compositor, comprising:
a first analyzer operating on one or more processors to perform statistical analysis on a first video sequence with an associated first electro-optical transfer function representing natural video content;
a second analyzer operating on the one or more processors to perform statistical analysis on a second video sequence with an associated electro-optical second transfer function representing synthetic video content;
a controller operating on the one or more processors, responsive to data representing output conditions, configured to:
convert one of the video sequences from an original dynamic range domain to a converted domain having a third electro-optical transfer function (EOTF) curve that matches a shape of the curve of the electro-optical transfer function associated with the other video sequence;
blend the converted video sequence with the other, non-converted video sequence; and
adjust at least a color component of the synthetic video component to match the native video component.
13. The video compositor of claim 12 , wherein the output condition is characteristics of the display device.
14. The video compositor of claim 12 , wherein the output condition is an ambient viewing condition about a display.
15. The video compositor of claim 12 , wherein the color component is a brightness level.
16. The video compositor of claim 12 , wherein the color component is a luminance level.
17. A video composition pipeline for processing a pair of video components, each from different sources and associated with different electro-optical transfer functions, a first video component having a higher dynamic range than the second video component, comprising:
a first data converter configured to convert the first video component having the higher dynamic range to a color space different from its original color space; and
a composition unit configured to:
convert the first video component to a third dynamic range, the third dynamic range being higher than the second dynamic range, wherein electro-optical transfer functions (EOTFs) associated with the converted first component and the second component are of complementary shapes;
assemble a video frame based on the converted first component and the second component; and
render an output frame based on the assembled video frame.
18. The video composition pipeline of claim 17 , further comprising a second data converter configured to convert the video component having the lower dynamic range to a color space different from its original color space.
19. The video composition pipeline of claim 17 , wherein the first component is of a higher dynamic range (HDR) and the second component is of an HDR different from the first component.
20. The video composition pipeline of claim 17 , wherein
the second component is of a standard dynamic range (sRGB) format and
a color space of the first component is of a range [−a, b], where a and b are each integer values, the range being programmable and greater than the sRGB format.
21. The video composition pipeline of claim 17 , wherein the assembly of the received components is based on ambient conditions including at least one of: a level of light leakage, a level of light reflection, a backlight level, a physical size of a display, a white point, an average pixel brightness of the display, and at least one color of light reflected by the display.
22. The video composition pipeline of claim 21 , wherein the level of light leakage is measured by a photo detector facing a viewer of the display.
23. The video composition pipeline of claim 21 , wherein, based on the ambient conditions, the assembly of the video frame includes at least one of: shading the frame, performing local tone mapping, and altering lookup tables for mapping one color space to another.
24. The video composition pipeline of claim 17 , wherein the evaluation of the received components is based on a mode of operation specifiable by a user, wherein the mode of operation overrides other bases for the assembly of the received components.
25. The video composition pipeline of claim 17 , wherein the assembly of the video frame is based on at least one position of a viewer relative to the display.