User interface virtualization for remote devices
View Patent ↗User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer's user interface style into the ergonomics of the other's. An agent running on a remote desktop collaborates with a corresponding client running on a client machine that accepts a “touch and swipe” style input. The agent and client exchange user interface metadata and user interface input events and translate the exchanged information to provide native graphical user interface elements (at the client machine) and simulated user actions (at the remote desktop). The agent running on the remote desktop may use an interface interaction API or library to programmatically manipulate the user interface of the remote desktop responsive and act as a proxy for the corresponding client.
1. A method of generating a local graphical user interface (GUI) on a client device that is connected to a server device having a remote GUI, the method comprising:
receiving, from the server device, a base image of the remote GUI and UI metadata describing a GUI element in the remote GUI;
generating the local GUI to be displayed on the touch screen of the client device, the local GUI including the base image received from the server device and a native GUI element to be displayed on the touch screen, the native GUI element generated according to the received UI metadata, the native GUI element corresponding to the GUI element in the remote GUI;
receiving, at the client device, a voice input through an input device of the client device;
processing the voice input to determine a command input to manipulate the remote GUI of the server device; and
transmitting, to the server device, information indicating the command input configured to manipulate the remote GUI.
2. The method of claim 1 , wherein the step of processing the voice input further comprises:
generating the command input based on a match of a plurality of rules associating a plurality of pre-determined commands with a corresponding input event to be executed at the server device.
3. The method of claim 1 , wherein:
the processing the voice input further comprises processing the voice input to determine a command input to manipulate the remote GUI the native GUI element, wherein the processing is performed based on information from the received UI metadata; and
the method further comprises transmitting, to the server device, information indicating that the corresponding GUI element in the remote GUI has been manipulated at the client device.
4. The method of claim 3 , wherein the step of processing the voice input further comprises:
determining the command input from a plurality of candidate texts based on the command input matching information from the UI metadata that specifies the GUI element of the remote GUI.
5. The method of claim 1 , wherein the UI metadata comprises semantic information related to contents of the GUI element in the remote GUI, wherein the UI metadata is organized in a tree-like structure.
6. The method of claim 1 , wherein the generated native GUI element is different than the corresponding GUI element in the remote GUI in at least one of size, shape, color, style, manner of interaction, animation, and interactive behavior.
7. The method of claim 1 , wherein the step of generating the native GUI element further comprises:
generating a system-provided control element based on the received UI metadata using a user interface framework local to the client device.
8. The method of claim 1 , wherein the local GUI element comprises the native GUI element arranged on top of the base image of the remote GUI received from the server device.
9. The method of claim 1 , further comprising:
receiving, at the client device, a touch input through the local GUI; and
responsive to determining that the native GUI element has been manipulated through the received input, transmitting, to the server device, information indicating that the corresponding GUI element in the remote GUI has been manipulated at the client device.
10. A client device system for generating a local graphical user interface (GUI), the client device system comprising:
a touch screen configured to display graphical data;
an input device configured to receive input data;
a processor; and
a memory storing instructions that, when executed by the processor cause the processor to perform a method, the method comprising:
receiving, from a server device connected to the client device, a base image of the remote GUI and UI metadata describing a GUI element in the remote GUI;
generating the local GUI to be displayed on the touch screen of the client device, the local GUI including the base image received from the server device and a native GUI element to be displayed on the touch screen, the native GUI element generated according to the received UI metadata, the native GUI element corresponding to the GUI element in the remote GUI;
receiving, at the client device, a voice input through the input device of the client device;
processing the voice input to determine a command input to manipulate the remote GUI of the server device; and
transmitting, to the server device, information indicating the command input configured to manipulate the remote GUI.
11. The client device system of claim 10 , wherein processing the voice input further comprises:
generating the command input based on a match of a plurality of rules associating a plurality of pre-determined commands with a corresponding input event to be executed at the server device.
12. The client device system of claim 10 , wherein:
the processing the voice input further comprises processing the voice input to determine a command input to manipulate the remote GUI the native GUI element, wherein the processing is performed based on information from the received UI metadata; and
the method further comprises transmitting, to the server device, information indicating that the corresponding GUI element in the remote GUI has been manipulated at the client device.
13. The client device system of claim 12 , wherein processing the voice input further comprises:
determining the command input from a plurality of candidate texts based on the command input matching information from the UI metadata that specifies the GUI element of the remote GUI.
14. The client device system of claim 10 , wherein the UI metadata comprises semantic information related to contents of the GUI element in the remote GUI, wherein the UI metadata is organized in a tree-like structure.
15. The client device system of claim 10 , wherein the generated native GUI element is different than the corresponding GUI element in the remote GUI in at least one of size, shape, color, style, manner of interaction, animation, and interactive behavior.
16. The client device system of claim 10 , wherein generating the native GUI element further comprises:
generating a system-provided control element based on the received UI metadata using a user interface framework local to the client device.
17. The client device system of claim 10 , wherein the local GUI element comprises the native GUI element arranged on top of the base image of the remote GUI received from the server device.
18. The client device system of claim 10 , wherein the method further comprises:
receiving, at the client device, a touch input through the local GUI; and
responsive to determining that the native GUI element has been manipulated through the received input, transmitting, to the server device, information indicating that the corresponding GUI element in the remote GUI has been manipulated at the client device.
19. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of generating a local graphical user interface (GUI) on a client device that is connected to a server device having a remote GUI, the method comprising:
receiving, from the server device, a base image of the remote GUI and UI metadata describing a GUI element in the remote GUI;
generating the local GUI to be displayed on the touch screen of the client device, the local GUI including the base image received from the server device and a native GUI element to be displayed on the touch screen, the native GUI element generated according to the received UI metadata, the native GUI element corresponding to the GUI element in the remote GUI;
receiving, at the client device, a voice input through an input device of the client device;
processing the voice input to determine a command input to manipulate the remote GUI of the server device; and
transmitting, to the server device, information indicating the command input configured to manipulate the remote GUI.
20. The non-transitory computer-readable medium of claim 19 , wherein the step of processing the voice input further comprises:
generating the command input based on a match of a plurality of rules associating a plurality of pre-determined commands with a corresponding input event to be executed at the server device.