Combined HDR/LDR video streaming
The invention provides methods for broadcasting video in a dual HDR/LDR format such that the video can be displayed in real time by both LDR and HDR display devices. Methods and devices of the invention process streams of pixels from multiple sensors in a frame-independent manner to produce an HDR video signal in real time. That HDR video signal is then tone-mapped to produce an LDR video signal, the LDR signal is subtracted from the HDR signal to calculate a residual signal, and the LDR signal and the residual signal are merged into a combined signal that is broadcast via a communications network.
1. A method for streaming video for high dynamic range (HDR) and low dynamic range (LDR) display, the method comprising:
receiving frame-independent streams of pixel values from multiple respective image sensors onto a processing device of an HDR video camera;
streaming the pixel values through a pipeline on the process device, wherein the pipeline includes
a synchronization module that synchronizes the streams of pixel values,
a kernel operation that identifies saturated pixel values, and
a merge operation that merges the pixel values to produce an HDR signal;
processing the HDR signal to produce an output signal comprising an LDR signal and a residual signal; and
transmitting the output signal for HDR and LDR display.
2. The method of claim 1 , wherein processing the HDR signal comprises tone-mapping the HDR video to produce the LDR signal and subtracting the LDR signal from the HDR signal to produce the residual signal.
3. The method of claim 2 , further comprising compressing the LDR video using an MPEG encoder.
4. The method of claim 1 , wherein the output signal is displayable as an 8-bit video by an LDR display, and displayable as HDR video by an HDR display.
5. The method of claim 1 , wherein the synchronization module contains line buffers that circulate early-arriving pixel values and release the early-arriving pixel values simultaneous with corresponding later-arriving pixel values.
6. The method of claim 5 , the multiple image sensors all capture images simultaneously through a single lens.
7. The method of claim 6 , further comprising receiving incoming light through the lens and splitting the light via at least one beamsplitter onto the multiple image sensors, wherein at least 95% of the light gathered by the imaging lens is captured by the multiple image sensors.
8. The method of claim 6 , wherein the multiple image sensors include at least a first sensor and a second sensor, wherein the first sensor receives a higher exposure than the second sensor, and wherein merging the sequences includes using first sensor pixel values that are not saturated and second sensor pixel values corresponding to the saturated pixel values.
9. The method of claim 6 , further comprising beginning to merge portions of the pixel values while still streaming later-arriving pixel values through the kernel operation.
10. The method of claim 9 , wherein the processing device is selected from the group consisting of a field-programmable gate array and an application-specific integrated circuit, and further wherein each of the image sensors comprises a color filter array, and the method further comprises demosaicing the HDR signal after the merging.
11. The method of claim 10 , wherein multiple image sensors capture images that are optically identical except for light level.