User interface virtualization profiles for accessing applications on remote devices
View Patent ↗User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer's user interface style into the ergonomics of the other's. An agent running on the remote desktop may use an interface interaction API or software framework to programmatically manipulate the user interface of the remote desktop and act as a proxy for a corresponding client running on a client device that accepts a “touch and swipe” style input. A user interface virtualization profile is used to specify application-specific augmentations and application-specific support for the interface interaction API.
1. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:
launching an application in the remote desktop of the server device, wherein the application includes one or more elements of a graphical user interface (GUI);
retrieving a user interface virtualization profile associated with the application;
generating user interface (UI) metadata that specifies the GUI of the application, wherein the UI metadata is generated according to the user interface virtualization profile and using an application programming interface (API) configured to programmatically manipulate the GUI of the application;
transmitting a base image of the GUI and the generated UI metadata to a touch input client device communicatively connected to the server device;
receiving, from the touch input client device, an input event indicating manipulation of the GUI of the application; and
invoking the API to programmatically manipulate the GUI of the application according to the received input event and further according to the user interface virtualization profile,
wherein the user interface virtualization profile associated with the application specifies application-specific support for the API configured to programmatically manipulate the GUI of the application.
2. The method of claim 1 , further comprising:
receiving, from the touch input client device, usage data associated with user interactions with the GUI of the application; and
modifying the user interface virtualization profile associated with the application based on the received usage data.
3. The method of claim 1 , wherein the user interface virtualization profile specifies a call to a desktop-side procedure comprised of one or more calls to the API to generate the UI metadata corresponding to the GUI of the application.
4. The method of claim 1 , wherein the user interface virtualization profile specifies a call to a desktop-side procedure comprised of one or more calls to the API to programmatically manipulate the GUI of the application responsive to a received input event.
5. The method of claim 1 , wherein the user interface virtualization profile comprises a federated profile retrieved from at least one of a master user interface virtualization profile server and a community-based user interface virtualization profile server.
6. The method of claim 1 , wherein the generating the UI metadata further comprises, responsive to detecting a selection of a shape object in a document for the application, generating UI metadata that specifies a position and size of the selected shape object based on the user interface virtualization profile; and
wherein the invoking the API of the guest operating system further comprises, responsive to receiving the input event, invoking the API to programmatically manipulate the position and size of the selected shape object based on the input event and the user interface virtualization profile.
7. The method of claim 1 , wherein the generating the UI metadata further comprises generating the UI metadata that specifies a drawing tool GUI for the application based on the user interface virtualization profile; and
responsive to receiving the input event, invoking the API to programmatically manipulate the drawing tool GUI for the application based on the input event and the user interface virtualization profile.
8. A non-transitory computer-readable storage medium comprising instructions that, when executed in a computing device, provide access to a remote desktop of a server device having a guest operating system by performing the steps of:
launching an application in the remote desktop of the server device, wherein the application includes one or more elements of a graphical user interface (GUI);
retrieving a user interface virtualization profile associated with the application;
generating user interface (UI) metadata that specifies the GUI of the application, wherein the UI metadata is generated according to the user interface virtualization profile and using an application programming interface (API) configured to programmatically manipulate the GUI of the application;
transmitting a base image of the GUI and the generated UI metadata to a touch input client device communicatively connected to the server device;
receiving, from the touch input client device, an input event indicating manipulation of the GUI of the application; and
invoking the API to programmatically manipulate the GUI of the application according to the received input event and further according to the user interface virtualization profile,
wherein the user interface virtualization profile associated with the application specifies application-specific support for the API configured to programmatically manipulate the GUI of the application.
9. The non-transitory computer-readable storage medium of claim 8 , further comprising:
receiving, from the touch input client device, usage data associated with user interactions with the GUI of the application; and
modifying the user interface virtualization profile associated with the application.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the user interface virtualization profile specifies a call to a desktop-side procedure comprised of one or more calls to the API to generate the UI metadata corresponding to the GUI of the application.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the user interface virtualization profile specifies a call to a desktop-side procedure comprised of one or more calls to the API of the guest operating system to programmatically manipulate the GUI of the application responsive to a received input event.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the user interface virtualization profile comprises a federated profile retrieved from at least one of a master user interface virtualization profile server and a community-based user interface virtualization profile server.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the step of generating the UI metadata further comprises, responsive to detecting a selection of a shape object in a document for the application, generating UI metadata that specifies a position and size of the selected shape object based on the user interface virtualization profile; and
wherein the step of invoking the API further comprises, responsive to receiving the input event, invoking the API to programmatically manipulate the position and size of the selected shape object based on the input event and the user interface virtualization profile.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the step of generating the UI metadata further comprises generating the UI metadata that specifies a drawing tool GUI for the application based on the user interface virtualization profile; and
wherein the step of invoking further comprises, responsive to receiving the input event, invoking the API to programmatically manipulate the drawing tool GUI for the application based on the input event and the user interface virtualization profile.
15. A computer system for providing access to a remote desktop, the computer system comprising a system memory and a processor programmed to carry out the steps of:
launching an application in the remote desktop of the computer system, wherein the application includes one or more elements of a graphical user interface (GUI);
retrieving a user interface virtualization profile associated with the application;
generating user interface (UI) metadata that specifies the GUI of the application, wherein the UI metadata is generated according to the user interface virtualization profile and using an application programming interface (API) configured to programmatically manipulate the GUI of the application;
transmitting a base image of the GUI and the generated UI metadata to a touch input client device communicatively connected to the computer system;
receiving, from the touch input client device, an input event indicating manipulation of the GUI of the application; and
invoking the API to programmatically manipulate the GUI of the application according to the received input event and further according to the user interface virtualization profile,
wherein the user interface virtualization profile associated with the application specifies application-specific support for the API configured to programmatically manipulate the GUI of the application.
16. The computer system of claim 15 , wherein the user interface virtualization profile specifies a call to a desktop-side procedure comprised of one or more calls to the API to generate the UI metadata corresponding to the GUI of the application.
17. The computer system of claim 15 , wherein the user interface virtualization profile specifies a call to a desktop-side procedure comprised of one or more calls to the API to programmatically manipulate the GUI of the application responsive to a received input event.