Decomposed multi-stream (DMS) techniques for video display systems
This disclosure describes techniques that can improve the generation of a decomposed multi-stream (DMS) by a host device of a video display system and the display of a DMS by a client device of the video display system. The techniques may apply different frame rates to different streams within a DMS, and the frame rates may depend on the content. For example, one stream within a DMS may comprise a sequence of full-motion video information, which may be rendered at a relatively high frame rate. However, another stream within the DMS may be associated with a background of the display, various graphic user interface control windows or elements, or a display window that includes non-video content (such as e-mail or a document). The second stream in the DMS may be rendered at a much slower frame rate than that used for the sequence of full-motion video information.
1. A method comprising:
generating a decomposed multi-stream (DMS) via a host device of a video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;
generating information in the DMS that identifies which of the first content or the second content is on top in the overlapping area;
communicating the DMS from the host device to a client device of the video display system; and
adjusting the DMS to only include one of the first content or the second content in response to user input.
2. The method of claim 1 , further comprising dynamically adjusting the first video frame rate and the second video frame rate in the DMS based on the first content and the second content.
3. The method of claim 1 , wherein generating the DMS includes intercepting the first content from an application and capturing the second content from a display buffer in the host device.
4. The method of claim 3 , further comprising dynamically reducing a capture rate associated with capturing the second content from the display buffer based on the first content and the second content.
5. The method of claim 1 , wherein generating the DMS includes encoding the second content.
6. The method of claim 1 , wherein generating the DMS includes reducing or eliminating data for either the first content or the second content in the overlapping area, wherein the reduced or eliminated data is below other data in the overlapping area.
7. The method of claim 1 , wherein generating the DMS includes intercepting the first content or the second content from an application.
8. The method of claim 7 , further comprising:
determining that intercepting the first content or the second content from the application is not possible; and
generating a signal composed stream (SCS) that includes the first content and the second content directly from a display buffer at the host device in response to determining that intercepting the first content or the second content from the application is not possible.
9. The method of claim 1 , wherein the first content comprises a video sequence and the second content comprises output of a non-video application.
10. The method of claim 9 , wherein the first video frame rate is greater than the second video frame rate.
11. The method of claim 1 , wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.
12. The method of claim 1 , further comprising:
determining a bandwidth available between the host device and the client device; and
adjusting one or both of the first video frame rate and the second video frame rate based on the bandwidth available.
13. The method of claim 1 , further comprising:
determining a bandwidth available between the host device and the client device; and
dynamically encoding one or both of the first content and the second content based on the bandwidth available.
14. The method of claim 1 , further comprising dynamically encoding one or both of the first content and the second content.
15. A method comprising:
receiving a decomposed multi-stream (DMS) at a client device of a video display system from a host device of the video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;
rendering the first display content and the second display content on the client device; and
rendering only one of the first content or the second content in response to user input.
16. The method of claim 15 , wherein the first content comprises a video sequence and the second content comprises output of a non-video application.
17. The method of claim 16 , wherein the first video frame rate is greater than the second video frame rate.
18. The method of claim 15 , wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.
19. The method of claim 15 , further comprising:
buffering the first content differently than the second content in the client device.
20. The method of claim 19 , further comprising applying different latency rules for display of the first content by the client device and display of the second content by the client device.
21. A host device of a video display system, the host device comprising:
a decomposed multi-stream (DMS) generator that generates a DMS, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate, and wherein the DMS generator generates information in the DMS that identifies which of the first content or the second content is on top in the overlapping area and adjusts the DMS to only include one of the first content or the second content in response to user input; and
a transport unit that communicates the DMS from the host device to a client device of the video display system.
22. The host device of claim 21 , wherein the DMS generator dynamically adjusts the first video frame rate and the second video frame rate in the DMS based on the first content and the second content.
23. The host device of claim 21 , wherein the DMS generator intercepts the first content from an application and captures the second content from a display buffer in the host device.
24. The host device of claim 23 , wherein the DMS generator dynamically reduces a capture rate associated with capturing the second content from the display buffer based on the first content and the second content.
25. The host device of claim 23 , wherein the DMS generator uses an encoder to encode the second content in the DMS.
26. The host device of claim 21 , wherein the DMS generator reduces or eliminates data for either the first content or the second content in the overlapping area, wherein the reduced or eliminated data is below other data in the overlapping area.
27. The host device of claim 21 , wherein the DMS generator intercepts the first content or the second content from an application.
28. The host device of claim 27 , wherein the DMS generator:
determines that intercepting the first content or the second content from the application is not possible; and
generates a signal composed stream (SCS) that includes the first content and the second content directly from a display buffer at the host device in response to determining that intercepting the first content or the second content from the application is not possible.
29. The host device of claim 21 , wherein the first content comprises a video sequence and the second content comprises output of a non-video application.
30. The host device of claim 29 , wherein the first video frame rate is greater than the second video frame rate.
31. The host device of claim 21 , wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.
32. The host device of claim 21 , wherein the DMS generator:
determines a bandwidth available between the host device and the client device; and
adjusts one or both of the first video frame rate and the second video frame rate based on the bandwidth available.
33. The host device of claim 21 , wherein the DMS generator:
determines a bandwidth available between the host device and the client device; and
dynamically encodes one or both of the first content and the second content based on the bandwidth available.
34. The host device of claim 21 , wherein the DMS generator dynamically encodes one or both of the first content and the second content.
35. A client device of a video display system, the client device comprising:
a transport unit that receives a decomposed multi-stream (DMS) from a host device, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;
a display unit; and
a DMS render unit that renders the first display content and the second display content on the display unit, wherein the DMS render renders only one of the first content or the second content in response to user input.
36. The client device of claim 35 , wherein the first content comprises a video sequence and the second content comprises output of a non-video application.
37. The client device of claim 36 , wherein the first video frame rate is greater than the second video frame rate.
38. The client device of claim 35 , wherein the first content comprises a video sequence and the second content comprises a graphical user interface element.
39. The client device of claim 35 , wherein the display unit includes buffers that are configured to buffer the first content differently than the second content.
40. The client device of claim 35 , wherein the render unit is configured to apply different latency rules for display of the first content by the display unit, and display of the second content by the display unit.
41. A non-transitory computer-readable storage medium comprising instructions that upon execution in a processor of a host device of a video display system, cause the host device to:
generate a decomposed multi-stream (DMS), wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;
generate information in the DMS that identifies which of the first content or the second content is on top in the overlapping area;
communicate the DMS from a host device to a client device of the video display system; and
adjusts the DMS to only include one of the first content or the second content in response to user input.
42. A non-transitory computer-readable storage medium comprising instructions that upon execution in a processor of a client device of a video display system, cause the client device to:
upon receiving a decomposed multi-stream (DMS) from a host device of the video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate,
render the first display content and the second display content on the client device, and
render only one of the first content or the second content in response to user input.
43. A host device of a video display system, the host device comprising:
means for generating a decomposed multi-stream (DMS), wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;
means for generating information in the DMS that identifies which of the first content or the second content is on top in the overlapping area;
means for communicating the DMS from the host device to a client device of the video display system; and
means for adjusting the DMS to only include one of the first content or the second content in response to user input.
44. A client device of a video display system, the client device comprising:
means for receiving a decomposed multi-stream (DMS) from a host device of the video display system, wherein the DMS defines first content in a first area of a display window and second content in a second area of the display window, wherein the first area and the second area each include an overlapping area, wherein the DMS comprises information that identifies which of the first content or the second content is on top in the overlapping area, wherein the first content defines a first video frame rate in the DMS and the second display content defines a second video frame rate in the DMS, the first video frame rate being different than the second video frame rate;
means for rendering the first display content and the second display content on the client device; and
means for rendering only one of the first content or the second content in response to user input.