Method and system for identifying drawing primitives for selective transmission to a remote display
Remote desktop servers include a display encoder that maintains a secondary framebuffer that contains display data to be encoded and transmitted to a remote client display and a list of display primitives effectuating updated display data in the secondary framebuffer. The display encoder submits requests to receive the list of drawing primitives to a video adapter driver that receives and tracks drawing primitives that, when executed, update a primary framebuffer.
1. A method comprising:
requesting a framebuffer update from a video adapter driver;
traversing a data structure to identify one or more regions in the framebuffer marked as changed since a previous framebuffer update request, wherein the data structure has a plurality of entries, wherein each entry represents a region of the framebuffer having a plurality of pixels;
for each marked region, comparing pixel values in the marked region in the framebuffer with pixel values of a corresponding region of a secondary framebuffer;
in response to determining that the pixel values of a particular marked region are different, copying the pixel values in the particular region to the corresponding region of the secondary framebuffer; and
determining updated display data to transmit to a remote client terminal.
2. The method of claim 1 , further comprising updating the data structure based on the comparing.
3. The method of claim 2 , wherein the updated data structure represents the regions in the secondary framebuffer that have actual changed pixel values.
4. The method of claim 2 , wherein updating the data structure comprises clearing all marked entries in the data structure that are unchanged, indicating that no pixel values were actually in the corresponding region of the framebuffer.
5. The method of claim 1 , wherein the data structure is a two dimensional bit vector where each entry in the bit vector represents an N×N region of pixels of the framebuffer.
6. The method of claim 1 , wherein the data structure is a region quadtree, wherein each level of the region quadtree represents a higher resolution bit vector of 2 N ×2 N pixel blocks.
7. The method of claim 1 , wherein each pixel block of the framebuffer is successive subdivided into smaller sub-quadrants until each changed pixel is contained within a smallest sub-quadrant.
8. A method comprising:
transmitting display update data to a remote client terminal for display, comprising:
obtaining display information including: updated display data in a secondary framebuffer, an encoder data structure that indicates which regions of the secondary framebuffer include updated display data, and a queue that includes a list of drawing primitives defining basic drawing instructions, wherein the encoder data structure has a plurality of entries, wherein each entry represents a region of the secondary framebuffer having a plurality of pixels;
determining whether to transmit the list of drawing primitives or updated display data to the remote client terminal based on an evaluation of bandwidth usage; and
transmitting either drawing primitives or the updated display data to the remote client in response to the determination.
9. The method of claim 8 , wherein obtaining the display information includes receiving a sequence number corresponding to a last drawing primitive to update display data in the secondary framebuffer.
10. The method of claim 9 , wherein each entry in the queue comprises a sequence number and a corresponding drawing primitive.
11. The method of claim 8 , wherein the queue is a FIFO queue comprising a list of completed drawing primitives added to the queue upon execution of the drawing primitives into a primary framebuffer.
12. A system comprising:
one or more computers having a display encoder and a video adapter driver for transmitting display data to a remote client terminal, wherein the one or more computers are configured to perform operations comprising:
requesting a framebuffer update from a video adapter driver;
traversing a data structure to identify one or more regions in the framebuffer marked as changed since a previous framebuffer update request, wherein the data structure has a plurality of entries, wherein each entry represents a region of the framebuffer having a plurality of pixels;
for each marked region, comparing pixel values in the marked region in the framebuffer with pixel values of a corresponding region of a secondary framebuffer;
in response to determining that the pixel values of a particular marked region are different, copying the pixel values in the particular region to the corresponding region of the secondary framebuffer; and
determining updated display data to transmit to a remote client terminal.
13. The system of claim 12 , wherein the one or more computers are further configured to perform operations comprising updating the data structure based on the comparing.
14. The system of claim 13 , wherein the updated data structure represents the regions in the secondary framebuffer that have actual changed pixel values.
15. The system of claim 13 , wherein updating the data structure comprises clearing all marked entries in the data structure that are unchanged, indicating that no pixel values were actually in the corresponding region of the framebuffer.
16. The system of claim 12 , wherein the data structure is a two dimensional bit vector where each entry in the bit vector represents an N×N region of pixels of the framebuffer.
17. The system of claim 12 , wherein the data structure is a region quadtree, wherein each level of the region quadtree represents a higher resolution bit vector of 2 N ×2 N pixel blocks.
18. The system of claim 12 , wherein each pixel block of the framebuffer is successive subdivided into smaller sub-quadrants until each changed pixel is contained within a smallest sub-quadrant.
19. A system comprising:
one or more computers having a display encoder and a video adapter driver for transmitting display data to a remote client terminal, wherein the one or more computers are configured to perform operations comprising:
transmitting display update data to a remote client terminal for display, comprising:
obtaining display information including: updated display data in a secondary framebuffer, an encoder data structure that indicates which regions of the secondary framebuffer include updated display data, and a queue that includes a list of drawing primitives defining basic drawing instructions, wherein the encoder data structure has a plurality of entries, wherein each entry represents a region of the secondary framebuffer having a plurality of pixels;
determining whether to transmit the list of drawing primitives or updated display data to the remote client terminal based on an evaluation of bandwidth usage; and
transmitting either drawing primitives or the updated display data to the remote client in response to the determination.
20. The system of claim 19 , wherein obtaining the display information includes receiving a sequence number corresponding to a last drawing primitive to update display data in the secondary framebuffer.
21. The system of claim 20 , wherein each entry in the queue comprises a sequence number and a corresponding drawing primitive.
22. The system of claim 19 , wherein the queue is a FIFO queue comprising a list of completed drawing primitives added to the queue upon execution of the drawing primitives into a primary framebuffer.