Systems and methods for using screen sampling to detect display changes
A graphical user interface (GUI) is polled to identify updates for transmission to a client device. A generalized poll of the GUI is conducted at a first frequency, wherein the poll compares color values of each pixel in a general region of the GUI with a previously known color value for that pixel to detect changed pixel color values. A local region that is a subset of the general region is determined to have a greater likelihood of being updated, e.g., regions encompassing a location of mouse and/or keyboard activity. A localized poll is conducted at a second frequency of only the local region of the GUI.
1. A method, comprising:
generating, by a server device, a graphical user interface (GUI) for remoting to a client device over a network connection;
polling, at the server device, a first portion of the GUI at a first frequency to detect one or more display updates to the GUI that are to be transmitted to the client device; and
polling, at the server device, a second portion of the GUI at a second frequency to detect the one or more display updates to the GUI that are to be transmitted to the client device, wherein the second frequency is different from the first frequency;
wherein polling the first portion of the GUI and polling the second portion of the GUI is performed by a sampling agent configured to:
read a display buffer containing a representation of the GUI by accessing one or more application programming interfaces (APIs) of a guest operating system (GOS) on a virtual machine (VM) executing on the server device; and
compare a color value of a pixel in the first portion of the GUI of a current frame in the display buffer with the color value of said pixel of a previous frame in the display buffer to detect changed pixel color values, wherein the changed pixel values indicate the one or more display updates to the GUI.
2. The method of claim 1 , further comprising:
identifying the second portion of the GUI based on a location of a user input event, wherein the user input event includes at least one of: a mouse move, a mouse click, or a keyboard input;
wherein the second frequency is higher than the first frequency.
3. The method of claim 1 , further comprising:
identifying the second portion of the GUI based on historical changes that occurred within the second portion of the GUI.
4. The method of claim 1 , wherein the first portion of the GUI is a general region of the GUI and wherein the second portion of the GUI is a local region that is a subset of the general region.
5. The method of claim 1 , further comprising:
identifying a third portion of the GUI; and
polling the third portion of the GUI at a third frequency that is different from the first frequency and from the second frequency.
6. The method of claim 1 , further comprising:
generating a sampling map that identifies pixels contained in the second portion of the GUI, wherein the sampling map is used to perform the polling of the second portion of the GUI.
7. The method of claim 1 , further comprising:
detecting the one or more display updates to the GUI based on polling the first portion and polling the second portion; and
transmitting the one or more display updates to the client device over the network connection.
8. A server computing device, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the server computing device to perform the operations of:
generating a graphical user interface (GUI) for remoting to a client device over a network connection;
polling a first portion of the GUI at a first frequency to detect one or more display updates to the GUI that are to be transmitted to the client device; and
polling a second portion of the GUI at a second frequency to detect the one or more display updates to the GUI that are to be transmitted to the client device, wherein the second frequency is different from the first frequency;
wherein polling the first portion of the GUI and polling the second portion of the GUI is performed by a sampling agent configured to:
read a display buffer containing a representation of the GUI by accessing one or more application programming interfaces (APIs) of a guest operating system (GOS) on a virtual machine (VM) executing on the server computing device; and
compare a color value of a pixel in the first portion of the GUI of a current frame in the display buffer with the color value of said pixel of a previous frame in the display buffer to detect changed pixel color values, wherein the changed pixel values indicate the one or more display updates to the GUI.
9. The server computing device of claim 8 , wherein the memory further comprises instructions executed by the at least one processor to cause the server computing device to perform the operations of:
identifying the second portion of the GUI based on a location of a user input event, wherein the user input event includes at least one of: a mouse move, a mouse click, or a keyboard input;
wherein the second frequency is higher than the first frequency.
10. The server computing device of claim 8 , wherein the memory further comprises instructions executed by the at least one processor to cause the server computing device to perform the operations of:
identifying the second portion of the GUI based on historical changes that occurred within the second portion of the GUI.
11. The server computing device of claim 8 , wherein the first portion of the GUI is a general region of the GUI and wherein the second portion of the GUI is a local region that is a subset of the general region.
12. The server computing device of claim 8 , wherein the memory further comprises instructions executed by the at least one processor to cause the server computing device to perform the operations of:
identifying a third portion of the GUI; and
polling the third portion of the GUI at a third frequency that is different from the first frequency and from the second frequency.
13. A non-transitory computer readable storage medium comprising one or more sequences of instructions, the instructions when executed by one or more processors causing the one or more processors to execute the operations of:
generating, by a server device, a graphical user interface (GUI) for remoting to a client device over a network connection;
polling, at the server device, a first portion of the GUI at a first frequency to detect one or more display updates to the GUI that are to be transmitted to the client device; and
polling, at the server device, a second portion of the GUI at a second frequency to detect the one or more display updates to the GUI that are to be transmitted to the client device, wherein the second frequency is different from the first frequency; wherein polling the first portion of the GUI and polling the second portion of the GUI is performed by a sampling agent configured to:
read a display buffer containing a representation of the GUI by accessing one or more application programming interfaces (APIs) of a guest operating system (GOS) on a virtual machine (VM) executing on the server device; and
compare a color value of a pixel in the first portion of the GUI of a current frame in the display buffer with the color value of said pixel of a previous frame in the display buffer to detect changed pixel color values, wherein the changed pixel values indicate the one or more display updates to the GUI.
14. The non-transitory computer readable storage medium of claim 13 , further comprising instructions executed by one or more processors and causing the one or more processors to execute the operations of:
identifying the second portion of the GUI based on a location of a user input event, wherein the user input event includes at least one of: a mouse move, a mouse click, or a keyboard input;
wherein the second frequency is higher than the first frequency.
15. The non-transitory computer readable storage medium of claim 13 , further comprising instructions executed by one or more processors and causing the one or more processors to execute the operations of:
identifying the second portion of the GUI based on historical changes that occurred within the second portion of the GUI.
16. The non-transitory computer readable storage medium of claim 13 , wherein the first portion of the GUI is a general region of the GUI and wherein the second portion of the GUI is a local region that is a subset of the general region.
17. The non-transitory computer readable storage medium of claim 13 , further comprising instructions executed by one or more processors and causing the one or more processors to execute the operations of:
detecting the one or more display updates to the GUI based on polling the first portion and polling the second portion; and
transmitting the one or more display updates to the client device over the network connection.