Display management for native user experiences
In some remote application execution environments, a client device (e.g., to which application output is sent and where user interaction is performed) may include native control elements for touch input such as virtual keyboards, virtual pickers and the like. When the server device detects an editable/control field of a remoted application receiving focus, the server may instruct the client to activate a native control element for user interaction. The client may determine the manner in which to modify an application display to resolve potential display conflicts. For example, a native control element may obscure or hide portions of the remoted application on the client display. Modifying the application display may include panning the display, zooming into or out of portions of the display, adapting the display for scrolling, and the like. The remote computing device may also transmit values for populating a native control element.
1. A method comprising:
receiving, by a client device, from a remote computing device that is executing a remoted application for the client device, data indicative of an application display for the remoted application;
generating, by the client device and based on the data indicative of the application display, the application display to allow a user to view the remoted application at the client device;
receiving, by the client device, from the remote computing device, a control value for an application control element associated with remoted application;
determining, by the client device, that a native graphical user input element will obscure one or more portions of the application display when the native graphical user input element is activated;
based on determining that the native graphical user input element will obscure the one or more portions of the application display, determining, by the client device, to perform a zoom out;
activating, by the client device, the native graphical user input element in association with the application display;
populating, by the client device, the native graphical user input element with the control value; and
modifying, by the client device, the application display by performing the zoom out such that the native graphical user input element is not obscuring the one or more portions of the application display.
2. The method of claim 1 , comprising:
receiving, by the client device, from the remote computing device, an instruction specifying a type of the application control element, and
wherein a type of the native graphical user input element corresponds to the type of the application control element.
3. The method of claim 1 , comprising:
receiving, by the client device, an interaction with the native graphical user input element; and
sending, by the client device and to the remote computing device, data that indicates the interaction.
4. The method of claim 3 , wherein the native graphical user input element comprises a menu with an item corresponding to the control value,
wherein the interaction comprises a selection of the item, and
wherein the data that indicates the interaction comprises an identification of the item.
5. The method of claim 1 , wherein the native graphical user input element comprises a virtual picker associated with an operating system of the client device, and wherein the virtual picker comprises the control value.
6. The method of claim 1 , comprising:
receiving a request to activate the native graphical user input element; and
activating, for a second time, the native graphical user input element.
7. The method of claim 1 , comprising:
receiving, by the client device, at least one of a size of the application control element or a position of the application control element.
8. The method of claim 1 , comprising:
sending, by the client device, first user input data that indicates the application control element is to receive focus; and
wherein determining that the native graphical user input element will obscure the one or more portions of the application display when the native graphical user input element is activated is performed based on determining that the native graphical user input element will obscure the application control element when the application control element receives focus.
9. The method of claim 8 , comprising:
sending, by the client device, second user input data that indicates focus has changed from application control element; and
after sending the second user input data, receiving, by the client device, a request to deactivate the native graphical user input element.
10. The method of claim 1 , comprising:
establishing communication between the client device and the remote computing device over a virtual channel; and
wherein receiving the control value is performed by receiving the control value via the virtual channel.
11. The method of claim 1 , comprising:
establishing communication between the client device and the remote computing device over a first virtual channel and a second virtual channel;
receiving, by the client device via the first virtual channel, remote control commands for the remoted application; and
receiving, by the client device via the second virtual channel, data that indicates one or more of an application window, a taskbar, or a system tray.
12. The method of claim 1 , comprising:
intercepting an event generated by a control within the remoted application; and
sending, to a window-management service of the remote computing device, data that indicates the event.
13. The method of claim 12 , comprising:
sending, to the remote computing device, one or more of a location coordinate for the control within the remoted application, a type of the control within the remoted application, a window handle of the control within the remoted application, or parent window information for the control within the remoted application.
14. One or more non-transitory computer-readable media storing executable instructions that, when executed, cause an apparatus to:
receive, from a remote computing device that is executing a remoted application for the apparatus, data indicative of an application display for the remoted application;
generate, based on the data indicative of the application display, the application display to allow a user to view the remoted application at the apparatus;
receive, from the remote computing device, a control value for an application control element associated with the remoted application;
determine that a native graphical user input element will obscure one or more portions of the application display when the native graphical user input element is activated;
based on determining that the native graphical user input element will obscure the one or more portions of the application display, determine to perform a zoom out;
activate the native graphical user input element in association with the application display;
populate the native graphical user input element with the control value; and
modify the application display by performing the zoom out such that the native graphical user input element is not obscuring the one or more portions of the application display.
15. An apparatus comprising:
one or more processors; and
memory storing executable instructions that, when executed by the one or more processors, cause the apparatus to:
receive, from a remote computing device that is executing a remoted application for the apparatus, data indicative of an application display for the remoted application;
generate, based on the data indicative of the application display, the application display to allow a user to view the remoted application at the apparatus;
receive, from the remote computing device, a control value for an application control element associated with the remoted application;
determine that a native graphical user input element will obscure one or more portions of the application display when the native graphical user input element is activated;
based on determining that the native graphical user input element will obscure the one or more portions of the application display, determine to perform a zoom out;
activate the native graphical user input element in association with the application display;
populate the native graphical user input element with the control value; and
modify the application display by performing the zoom out such that the native graphical user input element is not obscuring the one or more portions of the application display.
16. The apparatus of claim 15 , wherein the executable instructions, when executed by the one or more processors, cause the apparatus to:
receive an interaction with the native graphical user input element; and
send, to the remote computing device, data that indicates the interaction.
17. The apparatus of claim 15 , wherein the native graphical user input element comprises a virtual picker associated with an operating system of the apparatus, and wherein the virtual picker comprises the control value.
18. The one or more non-transitory computer-readable media of claim 14 , wherein the executable instructions, when executed, cause the apparatus to:
receive an interaction with the native graphical user input element; and
send, to the remote computing device, data that indicates the interaction.
19. The one or more non-transitory computer-readable media of claim 14 , wherein the native graphical user input element comprises a virtual picker associated with an operating system of the apparatus, and wherein the virtual picker comprises the control value.
20. The method of claim 1 , wherein the native graphical user input element is associated with an operating system of the client device.