Extending user interfaces of mobile apps to AR eyewear
An architecture is provided for packaging visual overlay-based user interfaces (UIs) into mobile device applications to work as user interface extensions that allow certain flows and logic to be displayed on an eyewear device when connected to the mobile device application. The extension of the UIs of the mobile device applications to the display of the eyewear device allows for inexpensive experimentation with augmented reality (AR) UIs for eyewear devices and allows for reusing of business logic across mobile devices and associated eyewear devices. For example, a mobile device application for maps or navigation may be extended to show directions on an associated eyewear device once the destination is chosen in the navigation application on the mobile device. In this example, the business logic would still live in the navigation application on the mobile device but the user would see AR directions overlaid on a display of the eyewear device.
1. A mobile device adapted to provide a user interface (UI) for a mobile device application executed thereon to an eyewear device, comprising:
a display;
a memory that stores instructions; and
a processor coupled to the display and the memory, wherein the processor executes the instructions to configure the mobile device to:
provide a wireless connection with the eyewear device;
load to the eyewear device, via the wireless connection, visual overlays adapted for the eyewear device for at least one user interface of the mobile device application;
enable a user to select a tethered mode for displaying on the eyewear device or on the mobile device and the eyewear device one or more of the loaded visual overlays and data bindings adapted for the eyewear device during execution of the mobile device application, the data bindings setting and updating data provided to the eyewear device; and
provide, via the wireless connection, a selection of at least one of the loaded visual overlays for overlay on a display of the eyewear device as an output of the mobile device application while the eyewear device remains in the tethered mode.
2. The mobile device as in claim 1 , further comprising eyewear UI extensions software including a background service that is executed by the processor to establish the wireless connection with the eyewear device.
3. The mobile device as in claim 2 , wherein the background service exposes a service to the mobile device application for communicating with the eyewear device via the wireless connection.
4. The mobile device as in claim 2 , wherein the mobile device application exposes a service of the eyewear UI extensions software to register itself as an eyewear extension.
5. The mobile device as in claim 4 , wherein the exposed service of the eyewear UI extensions software maintains a list of active registered mobile device applications.
6. The mobile device as in claim 1 , further comprising an operating system having an eyewear connectivity service that is executed by the processor to establish the wireless connection with the eyewear device.
7. The mobile device as in claim 6 , wherein the eyewear connectivity service is exposed to the mobile device application for communicating with the eyewear device via the wireless connection.
8. The mobile device as in claim 6 , wherein the mobile device application indicates that it contains an eyewear UI extension by declaring an eyewear category in its operating system manifest.
9. The mobile device as in claim 8 , wherein the eyewear connectivity service of the operating system maintains a list of active mobile device applications in the declared eyewear category.
10. The mobile device as in claim 1 , further comprising an eyewear Software Development Kit (SDK) that provides an interface for managing connectivity between the mobile device application and the eyewear device via the wireless connection.
11. The mobile device as in claim 10 , wherein the eyewear SDK provides layouts for the visual overlays that enable definition of layouts for the loaded visual overlays in a descriptive way and use of the data bindings to set and update data provided to the eyewear device via the wireless connection.
12. The mobile device as in claim 11 , wherein the layouts comprise XML files that are added to resources of the mobile device application.
13. The mobile device as in claim 1 , further comprising instructions that when executed by the processor configure the mobile device application to present an icon on the display indicating connectivity status of the mobile device and the eyewear.
14. A method of providing a user interface (UI) for a mobile device application executed on a mobile device to an eyewear device that communicates with the mobile device via a wireless connection, comprising:
establishing the wireless connection between the mobile device and the eyewear device;
loading from the mobile device to the eyewear device, via the wireless connection, visual overlays adapted for the eyewear device for at least one user interface of the mobile device application;
enabling a user to select a tethered mode for displaying on the eyewear device or on the mobile device and the eyewear device one or more of the loaded visual overlays and data bindings adapted for the eyewear device during execution of the mobile device application, the data bindings setting updating data provided to the eyewear device; and
providing, via the wireless connection, a selection of at least one of the loaded visual overlays for overlay on a display of the eyewear device as an output of the mobile device application while the eyewear device remains in the tethered mode.
15. The method of claim 14 , wherein establishing the wireless connection between the mobile device and the eyewear device comprises a background service of eyewear UI extensions software of the mobile device exposing a service to the mobile device application for communicating with the eyewear device via the wireless connection.
16. The method of claim 15 , further comprising the mobile device application registering itself as an eyewear extension using an exposed service of the eyewear UI extensions software and maintaining, by the exposed service of the eyewear UI extensions software, a list of active registered mobile device applications.
17. The method of claim 14 , wherein establishing the wireless connection between the mobile device and the eyewear device comprises an operating system of the mobile device establishing the wireless connection with the eyewear device by exposing an eyewear connectivity service of the operating system to the mobile device application that enables the mobile device application to communicate with the eyewear device via the wireless connection.
18. The method of claim 17 , further comprising the mobile device application indicating that it contains an eyewear UI extension by declaring an eyewear category in its operating system manifest and the eyewear connectivity service of the operating system maintaining a list of active mobile device applications in the declared eyewear category.
19. The method of claim 14 , further comprising providing an interface for managing connectivity between the mobile device application and the eyewear device via the wireless connection and adding XML files for the visual overlays to resources of the mobile device application, the XML files defining layouts for the loaded visual overlays in a descriptive way and using the data bindings to set and update data provided to the eyewear device via the wireless connection.
20. A non-transitory computer-readable storage medium that stores instructions that when executed by at least one processor cause the at least one processor to provide a user interface for a mobile device application executed on a mobile device to an eyewear device that communicates with the mobile device via a wireless connection by performing operations including:
establishing the wireless connection between the mobile device and the eyewear device;
loading from the mobile device to the eyewear device, via the wireless connection, visual overlays adapted for the eyewear device for at least one user interface of the mobile device application;
enabling a user to select a tethered mode for displaying on the eyewear device or on the mobile device and the eyewear device one or more of the loaded visual overlays and data bindings adapted for the eyewear device during execution of the mobile device application, the data bindings setting and updating data provided to the eyewear device; and
providing, via the wireless connection, a selection of at least one of the loaded visual overlays for overlay on a display of the eyewear device as an output of the mobile device application while the eyewear device remains in the tethered mode.