User interface virtualization techniques
View Patent ↗User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer's user interface style into the ergonomics of the other's. An agent running on a remote desktop exchanges metadata and input events with a corresponding client running on a client device that accepts a “touch and swipe” style input. The agent running on the remote desktop may use an interface interaction API or software framework to programmatically manipulate the user interface of the remote desktop and act as a proxy for the corresponding client. The client constructs and displays native graphical user interface elements at the client device that are virtual representations of a corresponding user interface element at the remote desktop, including file dialogs, clipboards, start menus, and task switchers.
1. A method for displaying a graphical user interface (GUI) on a client device connected to a server, the method comprising:
receiving, at the client device from the server, user interface (UI) metadata that describes elements of a GUI on a remote desktop of the server, wherein the UI metadata is received at the client device from a virtual machine (VM) running on the server;
receiving, at the client device from the server, a base image corresponding to the GUI on the remote desktop;
converting, at the client device, the UI metadata into at least one native GUI element that is native to the GUI of the client device;
displaying, at the client device, the base image and the native GUI element on the client device; and
responsive to detecting a manipulation of an element of the GUI of the client device, transmitting, by the client device to the server, an indication of a manipulation of a GUI element of the GUI of the remote desktop, wherein the indication corresponds to the manipulated element of the GUI of the client device.
2. The method of claim 1 , wherein:
the manipulation comprises a touch operation on a selection region that includes copy content; and
transmitting the indication of the manipulation includes transmitting a clipboard operation based on the selection region and the copy content.
3. The method of claim 1 , wherein the manipulation comprises:
manipulation of a native GUI element of the client device corresponding to a file dialog on the remote desktop.
4. The method of claim 3 , further comprising:
receiving UI metadata from the server corresponding to the file dialog; and
converting the UI metadata corresponding to the file dialog into a native file dialog.
5. The method of claim 1 , wherein the manipulation comprises:
manipulation of a native GUI element corresponding to a start menu UI element of the remote desktop.
6. The method of claim 1 , wherein the manipulation comprises:
manipulation of a native GUI element corresponding to a task switcher UI element of the remote desktop.
7. The method of claim 6 , further comprising:
receiving GUI metadata from the server corresponding to the task switcher; and
converting the GUI metadata corresponding to the task switcher to a native GUI element.
8. A method for providing a client device with access to a remote desktop on a server, the method comprising:
transmitting, from the server to the client device, user interface (UI) metadata that describes elements of a GUI on a remote desktop of the server, and that is to be converted at the client device into at least one native GUI element that is native to the GUI of the client device, wherein the UI metadata is transmitted to the client device from a virtual machine (VM) running on the server;
transmitting, from the server to the client device, a base image corresponding to the GUI on the remote desktop;
receiving, at the server from the client device, an indication of a manipulation of a GUI element of the GUI of the remote desktop, wherein the indication corresponds to the manipulated element of the GUI of the client device; and
interacting with the GUI on the remote desktop based on the manipulation.
9. The method of claim 8 , wherein:
the manipulation comprises a touch operation on a selection region that includes copy content; and
interacting with the GUI on the remote desktop includes performing a clipboard operation associated with the copy content.
10. The method of claim 8 , wherein the manipulation comprises:
the manipulation of manipulation of a native GUI element of the client device corresponding to a file dialog on the remote desktop.
11. The method of claim 10 , further comprising:
transmitting UI metadata corresponding to the file dialog to the client device.
12. The method of claim 8 , wherein the manipulation comprises:
manipulation of a native GUI element corresponding to a start menu UI element of the remote desktop.
13. The method of claim 8 , wherein the manipulation comprises:
manipulation of a native GUI element corresponding to a task switcher UI element of the remote desktop.
14. The method of claim 13 , further comprising:
transmitting GUI metadata corresponding to the task switcher to the client device.