System and method for synchronized streaming of a video-wall
View Patent ↗A system is disclosed for processing and streaming real-time graphics by a video-server for synchronized output via secondary-network connected display adapters to multiple displays arranged as a video-wall. This system enables the video-server to leverage performance advantages afforded by advanced GPUs, combined with low-cost Smart displays or System-on-Chip devices to deliver advanced realtime video-wall capabilities over the network while offering flexibility in the selection of network display adapters and still achieving synchronized output of multiple sub-image streams to selected end-point displays. This has applications generally in the field of real-time multiple-display graphics distribution as well as specific applications in the field of network video-walls. A method and computer readable medium are also disclosed that operate in accordance with the system.
1. A system for processing and streaming sub-image portions of video content in real-time by a networked-linked video-server to multiple network-linked display adapters with network synchronized time-clocks for synchronized output to multiple displays in a video-wall, the system comprising:
at least one graphical application being configured to render a sequence of graphical images to a server framebuffer on the networked-linked video-server;
a processing module to extract and process from the server framebuffer, using a specified placement of the multiple displays in the video-wall, corresponding sub-image portions of the graphical images;
a streaming module being configured to separately encode and stream the sub-image portions with matching time-stamps to their corresponding network-linked display adapters; and
the multiple network-linked display adapters being configured to receive, decode and synchronously output the sub-image portions to corresponding individual displays within the video-wall using the matching time-stamps with timing offsets being applied that exceed the latency incurred by the receiving, decoding, and synchronously outputting.
2. The system of claim 1 , wherein the streamed sub-image portions are stored on the multiple network-linked display adapters for repeated playback without re-transferring from the network-linked video-server.
3. The system of claim 1 , wherein an encoding frame-rate or bit-depth of the streamed sub-image portions are varied based on changes in:
user-specified rates,
a processing load of the multiple network-linked display adapters,
a determined frame-rate required by the content,
a measured rate of change of the content,
a determined available network bandwidth, and/or
a determined available processing load of the server.
4. The system of claim 1 , wherein the timing offsets being applied is applied by one or more of the following means:
the network-linked display adapter adding a timing offset to the sub-image time stamps to determine a presentation time,
the network-linked video-server adding a timing offset to a transmitted sub-image time stamps, and/or
a network time differential between clock timings of the network-linked server and the clock timings of the network-linked display adapters being utilized to create the timing offset.
5. The system of claim 1 , wherein the local timing information utilized by the network-linked display adapters is a local clock synchronized by one of:
Network Time Protocol (NTP),
Windows Time Service (W32Time),
Precision Time Protocol (PTP),
Periodic time synchronization,
Berkeley algorithm,
Clock-sampling mutual network synchronization,
Cristian's algorithm,
Global Positioning System time synchronization,
Inter-range Instrumentation Group time codes,
Precision Time Protocol (PTP),
Reference broadcast synchronization (RBS),
Reference Broadcast Infrastructure Synchronization (RBIS),
Synchronous Ethernet,
Synchronization in ad-hoc wireless networks,
Data-Plane Time-synchronization Protocol,
Data arrival time.
6. The system of claim 1 , wherein the received sub-image portions are scaled by the network-linked display adapters to match an output resolution of the multiple displays prior to outputting to the multiple displays in a video-wall.
7. The system of claim 1 , wherein the server-framebuffer on the network-linked video-server resides in GPU memory on the network-linked video-server and the processing is being done by the GPU on the network-linked video-server.
8. The system of claim 1 , wherein a calculated timing information contains a latency offset for latency compensation which is derived from one or more of:
a user specified value,
a default value,
end-to-end latency measurements,
determined network latency,
a worst case decoding speed of the network-linked display adapter,
a worst case display speed of any display in the video wall,
database stored values based on closest matching known system configurations,
an adapted value based on user selections,
a dynamically adapted value that adapts with varying network conditions,
a dynamically adapted value that varies based on the encoding method, and/or
a dynamically adapted value based on measured latency between video-server and network-linked display adapter devices.
9. The system of claim 1 , where an individual network-linked display adapter can be any one of:
a System-on-a-Chip housed within a display
a Smart display,
a stand-alone network device attached to a display via a cable,
a stand-alone network device wirelessly communicating with a display,
a TV box,
a TV stick,
a network connected zero-client device,
a wireless display,
a Personal Computer,
a Tablet,
a Cell phone,
a Smart phone,
a Holographic display,
a light emitting diode (LED) display controller,
a LED display, or
a Video projector.
10. The system of claim 1 , wherein the individual displays in a video-wall can be any one of:
a video monitor,
a touch-screen display,
a front-projection display,
a rear-projection display,
an illuminated digital display,
a reflective digital display,
a LED display,
a video projector, or
a holographic display.
11. The system of claim 1 , wherein a process of pairing the multiple of network-linked display adapters to the network-linked video-server includes ones of:
a unique pairing code output by each display,
auto-detection of the network-linked display adapters by the network-linked video-server,
calibration of multiple displays into the video-wall via an interactive graphical user interface (GUI) that includes using icon representations of each of multiple displays showing their relative position and placement,
calibration of multiple displays into a video-wall via a server-attached automated camera scanning for unique content sent to each display.
12. The system of claim 1 , wherein the encoding of the sub-image portions includes ones of: H264, HEVC(H265), mjpeg, H263, MPEG4, Theora, 3GP, Windows Media, Quicktime, MPEG-4, VP6, VP8, VP9, MPEG1, MPEG2, MPEG-TS, MPEG-4, DNXHD, XDCAM, DV, DVCPRO, DVCPRO, DVCProHD, IMX, XDCAM HD, XDCAM HD422, XDCAM EX, JPEG, JPEG 2000, or PNG, and the transmission of the resulting streams is via one of these protocols: RTSP, Transmission Control Protocol (TCP), User Datagram Protocol (UDP), HTTP, HDS, MPEG-DASH, RTSP, RTP, RTCP, or RTMP.
13. The system of claim 1 , wherein the network-linked display adapter device is a computing device running an operating system and a configuring of the network-linked display adapter includes installing a software application designed to receive, decode the streamed sub-image portions and synchronously output them to multiple displays in the video wall.
14. The system of claim 1 , where the network linking the video server and the display adapters can be any one of:
a wide area network,
a local area network,
a wireless network,
an ad hoc network,
a PTP network,
an Ethernet network,
a network supplying power and signal in parallel,
a bluetooth network,
Wifi, or
802.11.
15. The system of claim 1 , wherein the streaming module is configured to stream both audio and video and with some of the network-linked display adapters being configured to output audio to a speaker device.
16. The system of claim 1 , wherein the system further includes video-servers comprises multiple video-servers with network-synchronized time-clocks, working together, each streaming to a different set of network-linked display adapters, enabling a larger canvas than would be possible with a single server.
17. The system of claim 1 where, the extracting of the sub-image portions by the processing module includes one or more of:
correction for monoscopic or stereoscopic VR video input,
bezel correction for output to monitors,
curvature correction for output to non-flat displays,
overlap for edge-blending for output to projectors,
keystone corrections for output to projectors,
color-space conversion to facilitate output to target network-linked display adapters,
rotation of sub-image portions to account for rotated displays,
color-correction to compensate for differences between displays, and/or
resizing or scaling of the content appropriate to the target output display size.
18. The system of claim 1 , wherein the network-linked display adapters process a network-based authentication process prior to the outputting.
19. A non-transitory, tangible, computer readable medium having thereupon instructions that, when executed by at least one processor for at least one video-server network-linked to multiple time-synchronized display adapters, for synchronized output to multiple displays in a video-wall, the at least one processor is configured to perform:
rendering, by at least one application, video data frames containing video-image pixel data at a designated resolution to a video-server framebuffer residing in a computer readable memory;
accessing of a stored mapping of a placement of multiple displays comprising a video-wall and extracting corresponding multiple sub-images from the video-server framebuffer according to said mapping; and
processing, encoding and streaming the multiple sub-images with matching time-stamps for synchronized output to multiple displays via the corresponding coupled network-linked display adapters being in communication with the at least one video-server;
the multiple time-synchronized display adapters being configured to receive, decode and synchronously output the sub-image portions to corresponding individual displays within the video-wall using the matching time-stamps with timing offsets being applied that exceed the latency incurred by the receiving, decoding, and synchronously outputting.
20. A method for processing and streaming sub-image portions of real-time graphics by a network-linked video-server to multiple network-linked and time synchronized display adapters for synchronized output to multiple displays in a video-wall, the method comprising the steps of:
rendering, by at least one graphical application being configured to render graphical image data to a server-framebuffer residing in a computer readable memory;
extracting sub-image portions of the graphical image data from the server-framebuffer corresponding to a stored specified placement of displays within the video-wall;
processing, sequencing, encoding with matching time-stamps, and streaming the sub-image portions with accompanying timing information to multiple corresponding network-linked display adapters;
receiving and decoding by multiple network-linked and time synchronized display adapters the streamed sub-image sequences; and
synchronously outputting by the multiple network-linked and time synchronized display adapters using the matching time-stamps with timing offset being applied to the corresponding displays in the video-wall the appropriate sub-images using the local and streamed timing information.