Video stream management for remote graphical user interfaces
Embodiments enable display updates other than a video stream in a graphical user interface (GUI) to be rendered, encoded, and transmitted exclusive of the video stream. A virtual machine generates a GUI that includes an encoded video stream and other display updates. A virtual graphics processing unit (VGPU) stack associated with the VM renders the other display updates of the GUI to a framebuffer. The rendered display updates are encoded and transmitted to a client for display. The encoded video stream, or a modified (e.g., reduced bit rate) version of the encoded video stream, may be transmitted to the client, such that the client can display the encoded video stream within the GUI. For example, the encoded video stream may be selectively transmitted to the client based on the performance capabilities of the client.
1. A method comprising:
executing a virtual machine (VM) that generates a user interface using a virtual graphics processing unit (VGPU) stack associated with the VM;
receiving, at the VGPU stack, display updates to the user interface comprising a video stream and one or more graphical display updates;
determining whether a client computing device supports receiving the video stream in a first communication channel separate from a second communication channel used to transmit the one or more graphical display updates;
in response to determining that the client computing device supports receiving the video stream in the first communication channel separate from the second communication channel used to transmit the one or more graphical display updates:
rendering, by the VGPU stack, the one or more graphical display updates to the user interface to a framebuffer while excluding the video stream such that the video stream is not decoded or rendered to the framebuffer,
encoding, the rendered one or more graphical display updates, and
separately transmitting, to the client computing device, the one or more graphical display updates rendered to the framebuffer on the second communication channel and the video stream on the first communication channel;
receiving, at the VGPU stack, other graphical display updates to the user interface;
determining that a session has been disconnected from the client computing device and reconnected to a second client computing device that does not support receiving the video stream in a third communication channel separate from a fourth communication channel used to transmit the other graphical display updates; and
in response to determining that the second client computing device does not support receiving the video stream in the third communication channel separate from the fourth communication channel used to transmit the other graphical display updates:
rendering, by the VGPU stack, the other graphical display updates to the user interface and the video stream to the framebuffer,
encoding the video stream along with other graphical display updates for sending to the second client computing device; and
transmitting, to the second client computing device, the encoded video stream along with the other graphical display updates on a single communication channel.
2. The method of claim 1 , further comprising:
receiving, from the client computing device, a request to render the video stream with the one or more graphical display updates; and
encoding the video stream along with other display updates for sending to the client computing device.
3. The method of claim 1 , further comprising transmitting control information associated with the video stream to the client computing device, wherein the control information includes one or more of the following: a dimension of the video stream, a position of the video stream within the user interface, and compositing information associated with the video stream and the user interface.
4. The method of claim 1 , further comprising:
receiving, by the client computing device, the video stream and the one or more graphical display updates;
decoding, by the client computing device, the video stream and the one or more graphical display updates; and
displaying, by the client computing device, the decoded video stream and the decoded one or more graphical display updates within a user interface representing a remote desktop provided by the VM.
5. The method of claim 1 , further comprising:
modifying one or more of the following: a video compression format of the video stream, a bit rate of the video stream, a frame rate of the video stream, and a resolution of the video stream,
wherein transmitting the video stream comprises transmitting the modified video stream.
6. A system comprising:
one or more processors; and
at least one memory, the one or more processors and the at least one memory configured to:
execute a virtual machine (VM) that generates a user interface using a virtual graphics processing unit (VGPU) stack associated with the VM;
receive, at the VGPU stack, display updates to the user interface comprising a video stream and one or more graphical display updates;
determine whether a client computing device supports receiving the video stream in a first communication channel separate from a second communication channel used to transmit the one or more graphical display updates; and
in response to determining that the client computing device supports receiving the video stream in the first communication channel separate from the second communication channel used to transmit the one or more graphical display updates:
render, by the VGPU stack, the one or more graphical display updates to the user interface to a framebuffer while excluding the video stream such that the video stream is not decoded or rendered to the framebuffer,
encode, the rendered one or more graphical display updates, and
separately transmit, to a client computing device, the one or more graphical display updates rendered to the framebuffer and the video stream;
receive, at the VGPU stack, other graphical display updates to the user interface;
determine that a session has been disconnected from the client computing device and reconnected to a second client computing device that does not support receiving the video stream in a third communication channel separate from a fourth communication channel used to transmit the other graphical display updates; and
in response to determining that the second client computing device does not support receiving the video stream in the third communication channel separate from the fourth communication channel used to transmit the other graphical display updates:
render, by the VGPU stack, the other graphical display updates to the user interface and the video stream to the framebuffer,
encode the video stream along with other graphical display updates for sending to the second client computing device; and
transmit, to the second client computing device, the encoded video stream along with the other graphical display updates on a single communication channel.
7. The system of claim 6 , wherein the one or more processors and the at least one memory are further configured to:
receive, from the client computing device, a request to render the video stream with the one or more graphical display updates; and
encode the video stream along with other display updates for sending to the client computing device.
8. The system of claim 6 , wherein the one or more processors and the at least one memory are further configured to:
transmit control information associated with the video stream to the client computing device, wherein the control information includes one or more of the following: a dimension of the video stream, a position of the video stream within the user interface, and compositing information associated with the video stream and the user interface.
9. The system of claim 6 , wherein the one or more processors and the at least one memory are further configured to:
receive, by the client computing device, the video stream and the one or more graphical display updates;
decode, by the client computing device, the video stream and the one or more graphical display updates; and
display, by the client computing device, the decoded video stream and the decoded one or more graphical display updates within a user interface representing a remote desktop provided by the VM.
10. The system of claim 6 , wherein the one or more processors and the at least one memory are further configured to:
modify one or more of the following: a video compression format of the video stream, a bit rate of the video stream, a frame rate of the video stream, and a resolution of the video stream,
wherein to transmit the video stream comprises to transmit the modified video stream.
11. One or more non-transitory computer-readable storage media having instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations comprising:
executing a virtual machine (VM) that generates a user interface using a virtual graphics processing unit (VGPU) stack associated with the VM;
receiving, at the VGPU stack, display updates to the user interface comprising a video stream and one or more graphical display updates;
determining whether a client computing device supports receiving the video stream in a first communication channel separate from a second communication channel used to transmit the one or more graphical display updates; and
in response to determining that the client computing device supports receiving the video stream in the first communication channel separate from the second communication channel used to transmit the one or more graphical display updates:
rendering, by the VGPU stack, the one or more graphical display updates to the user interface to a framebuffer while excluding the video stream such that the video stream is not decoded or rendered to the framebuffer,
encoding, the rendered one or more graphical display updates, and
separately transmitting, to a client computing device, the one or more graphical display updates rendered to the framebuffer and the video stream;
receiving, at the VGPU stack, other graphical display updates to the user interface;
determining that a session has been disconnected from the client computing device and reconnected to a second client computing device that does not support receiving the video stream in a third communication channel separate from a fourth communication channel used to transmit the other graphical display updates; and
in response to determining that the second client computing device does not support receiving the video stream in the third communication channel separate from the fourth communication channel used to transmit the other graphical display updates:
rendering, by the VGPU stack, the other graphical display updates to the user interface and the video stream to the framebuffer,
encoding the video stream along with other graphical display updates for sending to the second client computing device; and
transmitting, to the second client computing device, the encoded video stream along with the other graphical display updates on a single communication channel.
12. The one or more non-transitory computer-readable storage media of claim 11 , further comprising:
receiving, from the client computing device, a request to render the video stream with the one or more graphical display updates; and
encoding the video stream along with other display updates for sending to the client computing device.
13. The one or more non-transitory computer-readable storage media of claim 11 , further comprising transmitting control information associated with the video stream to the client computing device, wherein the control information includes one or more of the following: a dimension of the video stream, a position of the video stream within the user interface, and compositing information associated with the video stream and the user interface.
14. The one or more non-transitory computer-readable storage media of claim 11 , further comprising:
receiving, by the client computing device, the video stream and the one or more graphical display updates;
decoding, by the client computing device, the video stream and the one or more graphical display updates; and
displaying, by the client computing device, the decoded video stream and the decoded one or more graphical display updates within a user interface representing a remote desktop provided by the VM.
15. The one or more non-transitory computer-readable storage media of claim 11 , further comprising:
modifying one or more of the following: a video compression format of the video stream, a bit rate of the video stream, a frame rate of the video stream, and a resolution of the video stream,
wherein transmitting the video stream comprises transmitting the modified video stream.