Controller graphical user interface based on interaction data
Techniques for controller graphical user interface based on interaction data are described and are implementable to generate a controller graphical user interface for display by a first device to control digital content displayed on a second device. For instance, an application such as a mobile gaming application is displayed on a mobile device and digital content from the application is communicated for display by a display device as part of a content connectivity session. One or more control regions of the application are determined based on a monitored interaction with a user interface of the mobile device. One or more control graphics are identified that correlate to the one or more control regions. A controller graphical user interface is generated that displays the one or more control graphics and filters out extraneous digital content. The controller graphical user interface is then displayed by the mobile device.
1. A computing device, comprising:
a content control module implemented at least partially in hardware and configured to:
establish a content connectivity session between the computing device and a remote display device;
display digital content from a gaming application on the computing device and communicate the digital content for display by the remote display device as part of the content connectivity session;
determine, via one or more touch sensors of the computing device, one or more control regions of the gaming application based on a frequency of touch of a user as part of a monitored interaction with the gaming application;
segment the digital content from the gaming application into constituent visual features;
calculate a centroid for each of the constituent visual features and an epicenter of each control region of the one or more control regions;
identify one or more control graphics by correlating one or more of the centroids to one or more of the epicenters;
generate a controller graphical user interface by filtering the digital content to display the one or more control graphics and suppress display of portions of the digital content extraneous to the one or more control graphics; and
display the controller graphical user interface on the computing device and cause the gaming application to be displayed on the remote display device.
2. The computing device as described in claim 1 , wherein the content connectivity session is established automatically and without user intervention responsive to the gaming application being launched by the computing device.
3. The computing device as described in claim 1 , wherein the one or more control regions are determined based in part on saved data associated with a user of the computing device based on a previous content connectivity session.
4. The computing device as described in claim 1 , wherein the one or more control graphics are identified in part using image recognition techniques to identify one or more candidate visual features that likely correspond to the one or more control regions.
5. The computing device as described in claim 1 , wherein to generate the controller graphical user interface includes capturing a screenshot of the digital content and filtering out regions of the screenshot that do not correspond to the one or more control graphics.
6. The computing device as described in claim 1 , wherein the content control module is further configured to update a location of the one or more control graphics relative to the controller graphical user interface based on a subsequent monitored interaction.
7. The computing device as described in claim 1 , wherein the one or more control graphics are determined based in part on a configuration session that includes receiving user input to the computing device to identify the one or more control regions.
8. A method, comprising:
displaying an application depicting digital content on a mobile device and communicating the digital content for display by a remote display device as part of a content connectivity session;
determining one or more control regions of the application based on a monitored interaction with a user interface of the mobile device via one or more touch sensors of the mobile device;
identifying one or more control graphics by:
segmenting the digital content into constituent visual features;
calculating an epicenter of each control region of the one or more control regions and a centroid of each of the constituent visual features; and
identifying centroids of the constituent visual features that correspond to the epicenters of the one or more control regions;
generating a controller graphical user interface that displays the one or more control graphics and suppresses display of portions of the digital content extraneous to the one or more control graphics; and
displaying the controller graphical user interface on the mobile device and the application on the remote display device.
9. The method as described in claim 8 , wherein the content connectivity session is established automatically and without user intervention responsive to the application being launched by the mobile device.
10. The method as described in claim 8 , wherein the one or more control regions are determined based on a frequency of touch by a user that is a subject of the monitored interaction.
11. The method as described in claim 8 , wherein the one or more control regions are determined based in part on saved data associated with a user of the mobile device based on a previous content connectivity session.
12. The method as described in claim 8 , wherein the one or more control graphics are identified based in part using image recognition techniques to identify one or more candidate visual features that likely correspond to the one or more control regions.
13. The method as described in claim 8 , wherein generating the controller graphical user interface includes capturing a screenshot of the digital content and filtering out regions of the screenshot that do not correspond to the one or more control graphics.
14. The method as described in claim 8 , further comprising updating a location of the one or more control graphics relative to the controller graphical user interface based on a subsequent monitored interaction.
15. A system, comprising:
one or more processors; and
one or more computer-readable storage media storing instructions that are executable by the one or more processors to:
establish a content connectivity session between a mobile device and a remote display device;
communicate digital content from a gaming application to the remote display device for display;
generate a controller graphical user interface that corresponds to the gaming application for display by the mobile device as part of a content connectivity session, the controller graphical user interface including one or more control graphics that have centroids that correspond to epicenters of one or more control regions of the gaming application and suppresses display of portions of the digital content extraneous to the one or more control graphics;
monitor an interaction with the controller graphical user interface displayed by the mobile device to detect one or more updated control regions;
generate an updated controller graphical user interface by editing the one or more control graphics in the controller graphical user interface based on the one or more updated control regions; and
display the updated controller graphical user interface on the mobile device.
16. The system as described in claim 15 , wherein to generate the updated controller graphical user interface includes editing a location of the one or more control graphics relative to the controller graphical user interface.
17. The system as described in claim 15 , wherein to generate the updated controller graphical user interface includes editing a size of the one or more control graphics based on the one or more updated control regions.
18. The computing device as described in claim 1 , wherein the one or more control regions are further determined based on a distribution of touch of the user as part of the monitored interaction.
19. The method as described in claim 8 , wherein the determining the one or more control regions includes generating a heat map based on one or more instances of proximity-based input to the mobile device as part of the monitored interaction.
20. The method as described in claim 8 , wherein the determining the one or more control regions includes implementing a control region algorithm that applies weights to each instance of proximity-based input detected over a period of time.