Split framebuffer encoding
A virtual desktop can be remoted to a client device by transmitting a video stream using a video coding format, such as H.264, where higher resolution video can be streamed than natively supported by the encoder or decoder. The encoder framebuffer is divided into a plurality of equal size child buffers, each child buffer is assigned an identifier for identifying the child buffer. Padding pixels can be added to the child buffers to make the child buffer size acceptable for the video coding format. The child buffers are encoded using the video coding format and the encoded frames are transmitted to a client. On the decoder side, the encoded child frames are decoded, padding pixels are removed, and the data is used to update corresponding child buffers in a framebuffer on the client display system.
1. A method for conveying a video stream to a client device from a server, comprising:
receiving the video stream containing encoded child frames at the client device from the server, each child frame containing screen data corresponding to one of two or more child buffers, wherein a framebuffer of the client device is divided into the two or more child buffers, and wherein the server is configured to:
receive a frame in a framebuffer on the server, the framebuffer on the server being divided into two or more child buffers corresponding to the two or more child buffers of the framebuffer of the client;
determine that a child buffer of the framebuffer on the server contains changed screen data from a preceding frame received in the framebuffer of the server;
in response to determining that the child buffer of the framebuffer on the server contains changed screen data, encode the determined changed child buffer to produce an encoded child frame and convey the encoded child frame to the client device in the video stream;
decoding, by a decoder on the client device, the received child frame in the video stream;
updating a corresponding child buffer of the framebuffer of the client device with the received child frame; and
displaying contents of the updated framebuffer of the client device on a display of the client device.
2. The method of claim 1 , further comprising determining to which child buffer the received child frame corresponds based on an identifier associated with the received child frame in the video stream that indicates to which child buffer of the two or more child buffers the received child frame corresponds.
3. The method of claim 2 , wherein the identifier comprises a row of blocks added to the received child frame that uniquely identifies the child buffer corresponding to the received child frame with color values of pixels in the added blocks.
4. The method of claim 1 , further comprising removing padding pixels from the received child frame, wherein the padding pixels are added to the received child frame to make the size of the received child frame suitable for the decoder.
5. The method of claim 1 , wherein the video stream transmits a graphical user interface of a virtual desktop from a host server to the client device in a virtual desktop infrastructure.
6. The method of claim 1 , wherein the framebuffer is divided into the two or more child buffers such that the resulting size of each child buffer is sufficiently small to be processed by the decoder.
7. The method of claim 6 , wherein the decoder is a hardware decoder.
8. A computing device for conveying a video stream to a client device from a server, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the computing device to perform the steps of:
receiving the video stream containing encoded child frames at the client device from the server, each child frame containing screen data corresponding to one of two or more child buffers, wherein a framebuffer of the client device is divided into the two or more child buffers, and wherein the server is configured to:
receive a frame in a framebuffer on the server, the framebuffer on the server being divided into two or more child buffers corresponding to the two or more child buffers of the framebuffer of the client;
determine that a child buffer of the framebuffer on the server contains changed screen data from a preceding frame received in the framebuffer of the server;
in response to determining that the child buffer of the framebuffer on the server contains changed screen data, encode the determined changed child buffer to produce an encoded child frame and convey the encoded child frame to the client device in the video stream;
decoding, by a decoder on the client device, the received child frame in the video stream;
updating a corresponding child buffer of the framebuffer of the client device with the received child frame; and
displaying contents of the updated framebuffer of the client device on a display of the client device.
9. The computing device of claim 8 , wherein the memory further includes instructions that when executed by the at least one processor, cause the computing device to perform the steps of determining to which child buffer the received child frame corresponds based on an identifier associated with the received child frame in the video stream that indicates to which child buffer of the two or more child buffers the received child frame corresponds.
10. The computing device of claim 9 , wherein the identifier comprises a row of blocks added to the received child frame that uniquely identifies the child buffer corresponding to the received child frame with color values of pixels in the added blocks.
11. The computing device of claim 8 , wherein the memory further includes instructions that when executed by the at least one processor, cause the computing device to perform the steps of removing padding pixels from the received child frame, wherein the padding pixels are added to the received child frame to make the size of the received child frame suitable for the decoder.
12. The computing device of claim 8 , wherein the video stream transmits a graphical user interface of a virtual desktop from a host server to the client device in a virtual desktop infrastructure.
13. The computing device of claim 8 , wherein the framebuffer is divided into the two or more child buffers such that the resulting size of each child buffer is sufficiently small to be processed by the decoder.
14. The computing device of claim 13 , wherein the decoder is a hardware decoder.
15. A non-transitory computer readable storage medium for conveying a video stream to a client device from a server comprising one or more sequences of instructions, the instructions when executed by one or more processors causing the one or more processors to execute the operations of:
receiving the video stream containing encoded child frames at the client device from the server, each child frame containing screen data corresponding to one of two or more child buffers, wherein a framebuffer of the client device is divided into the two or more child buffers, and wherein the server is configured to:
receive a frame in a framebuffer on the server, the framebuffer on the server being divided into two or more child buffers corresponding to the two or more child buffers of the framebuffer of the client;
determine that a child buffer of the framebuffer on the server contains changed screen data from a preceding frame received in the framebuffer of the server;
in response to determining that the child buffer of the framebuffer on the server contains changed screen data, encode the determined changed child buffer to produce an encoded child frame and convey the encoded child frame to the client device in the video stream;
decoding, by a decoder on the client device, the received child frame in the video stream;
updating a corresponding child buffer of the framebuffer of the client device with the received child frame; and
displaying contents of the updated framebuffer of the client device on a display of the client device.
16. The non-transitory computer readable storage medium of claim 15 , further comprising instructions that when executed by the one or more processors cause the one or more processors to execute the operations of determining to which child buffer the received child frame corresponds based on an identifier associated with the received child frame in the video stream that indicates to which child buffer of the two or more child buffers the received child frame corresponds.
17. The non-transitory computer readable storage medium of claim 16 , wherein the identifier comprises a row of blocks added to the received child frame that uniquely identifies the child buffer corresponding to the received child frame with color values of pixels in the added blocks.
18. The non-transitory computer readable storage medium of claim 15 , further comprising instructions that when executed by the one or more processors cause the one or more processors to execute the operations of removing padding pixels from the received child frame, wherein the padding pixels are added to the received child frame to make the size of the received child frame suitable for the decoder.
19. The non-transitory computer readable storage medium of claim 15 , wherein the video stream transmits a graphical user interface of a virtual desktop from a host server to the client device in a virtual desktop infrastructure.
20. The non-transitory computer readable storage medium of claim 15 , wherein the framebuffer is divided into the two or more child buffers such that the resulting size of each child buffer is sufficiently small to be processed by the decoder.