Coordinating adjustments to composite graphical user interfaces generated by multiple devices
In some implementations, a user device can coordinate adjustments to a composite GUI generated in cooperation with an accessory device that presents the composite graphical user interface. For example, the user device can receive configuration data that defines the maximum GUI display area (e.g., size, dimensions) of the accessory device, various view areas within the GUI display area of the accessory device into which the user device can render GUIs, and/or transitions between the various view areas. The user device can generate a display buffer based on the maximum GUI display area, generate a graphical user interface in an area of the buffer corresponding to a current view area specified by the accessory device, and send to the accessory device video frames corresponding to the display buffer. The accessory device can generate a composite GUI based on the received video frames that include the user device generated GUI.
1. A method comprising:
receiving, by a user device, a current view area definition from an accessory device, the current view area definition defining a first portion of a maximum GUI display area where the user device will generate a user device GUI;
generating, by the user device, video frames having dimensions corresponding to the maximum GUI display area and including a user device GUI in a second portion of the video frames corresponding to the first portion of the maximum GUI display area;
generating, by the user device, video frame metadata describing a size and location of the user device GUI within the video frames; and
sending, by the user device, the video frames and the video frame metadata to the accessory device, where the accessory device obtains the user device GUI from the video frames based on the metadata and presents the user device GUI on a display of the accessory device.
2. The method of claim 1 , further comprising:
receiving, by the user device, composite graphical user interface (GUI) configuration data from the accessory device, the composite GUI configuration data defining a maximum GUI display area and a plurality of view area definitions.
3. The method of claim 1 , further comprising:
generating, by the user device, an external display buffer having a size corresponding to the maximum GUI display area;
generating, by the user device, a user device GUI in a third portion of the external display buffer corresponding to the first portion of the maximum GUI display area; and
generating, by the user device, the video frames based on the external display buffer.
4. The method of claim 1 , further comprising:
receiving, by the user device, an update view area definition from the accessory device that identifies a fourth portion of the maximum GUI display area for generating the user device GUI; and
generating, by the user device, video frames having dimensions corresponding to the maximum GUI display area and including a user device GUI in a fifth portion of the video frames corresponding to the fourth portion of the maximum GUI display area.
5. The method of claim 4 , further comprising:
generating, by the user device, a transition animation for transitioning from the current view area to the updated view area by generating the user device GUI within intermediate view areas between the current view area and the updated view area within the video frames;
generating, by the user device, video frame metadata that describes a size and location of the intermediate view areas within the video frames; and
sending the video frames and the video frame metadata to the accessory device, where the accessory device uses the intermediate view areas and the metadata to generate a composite GUI that presents the transition animation.
6. The method of claim 4 , further comprising:
receiving, by the user device from the accessory device, user input related to the user device GUI presented by the accessory device;
determining, by the user device, a second view area based on the user input;
sending, by the user device, a request to transition to the second view area; and
receiving, by the user device, the updated view area definition that corresponds to the second view area.
7. The method of claim 1 , further comprising:
determining, by the user device, a safe area within the current view area based on the view area definition; and
determining, by the user device, at least one important feature of the user device GUI; and
generating, by the user device, the at least one important feature within the safe area of the current view area.
8. A non-transitory computer readable medium including one or more sequences of instructions that, when executed by one or more processors, cause the processors to perform operations comprising:
receiving, by a user device, a current view area definition from an accessory device, the current view area definition defining a first portion of a maximum GUI display area where the user device will generate a user device GUI;
generating, by the user device, video frames having dimensions corresponding to the maximum GUI display area and including a user device GUI in a second portion of the video frames corresponding to the first portion of the maximum GUI display area;
generating, by the user device, video frame metadata describing a size and location of the user device GUI within the video frames; and
sending, by the user device, the video frames and the video frame metadata to the accessory device, where the accessory device obtains the user device GUI from the video frames based on the metadata and presents the user device GUI on a display of the accessory device.
9. The non-transitory computer readable medium of claim 8 , wherein the instructions cause the processors to perform operations comprising:
receiving, by the user device, composite graphical user interface (GUI) configuration data from the accessory device, the composite GUI configuration data defining a maximum GUI display area and a plurality of view area definitions.
10. The non-transitory computer readable medium of claim 8 , wherein the instructions cause the processors to perform operations comprising:
generating, by the user device, an external display buffer having a size corresponding to the maximum GUI display area;
generating, by the user device, a user device GUI in a third portion of the external display buffer corresponding to the first portion of the maximum GUI display area; and
generating, by the user device, the video frames based on the external display buffer.
11. The non-transitory computer readable medium of claim 8 , wherein the instructions cause the processors to perform operations comprising:
receiving, by the user device, an update view area definition from the accessory device that identifies a fourth portion of the maximum GUI display area for generating the user device GUI; and
generating, by the user device, video frames having dimensions corresponding to the maximum GUI display area and including a user device GUI in a fifth portion of the video frames corresponding to the fourth portion of the maximum GUI display area.
12. The non-transitory computer readable medium of claim 11 , wherein the instructions cause the processors to perform operations comprising:
generating, by the user device, a transition animation for transitioning from the current view area to the updated view area by generating the user device GUI within intermediate view areas between the current view area and the updated view area within the video frames;
generating, by the user device, video frame metadata that describes a size and location of the intermediate view areas within the video frames; and
sending the video frames and the video frame metadata to the accessory device, where the accessory device uses the intermediate view areas and the metadata to generate a composite GUI that presents the transition animation.
13. The non-transitory computer readable medium of claim 11 , wherein the instructions cause the processors to perform operations comprising:
receiving, by the user device from the accessory device, user input related to the user device GUI presented by the accessory device;
determining, by the user device, a second view area based on the user input;
sending, by the user device, a request to transition to the second view area; and
receiving, by the user device, the updated view area definition that corresponds to the second view area.
14. The non-transitory computer readable medium of claim 8 , wherein the instructions cause the processors to perform operations comprising:
determining, by the user device, a safe area within the current view area based on the view area definition; and
determining, by the user device, at least one important feature of the user device GUI; and
generating, by the user device, the at least one important feature within the safe area of the current view area.
15. A system comprising:
one or more processors, and
a non-transitory computer readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the processors to perform operations comprising:
receiving, by a user device, a current view area definition from an accessory device, the current view area definition defining a first portion of a maximum GUI display area where the user device will generate a user device GUI;
generating, by the user device, video frames having dimensions corresponding to the maximum GUI display area and including a user device GUI in a second portion of the video frames corresponding to the first portion of the maximum GUI display area;
generating, by the user device, video frame metadata describing a size and location of the user device GUI within the video frames; and
sending, by the user device, the video frames and the video frame metadata to the accessory device, where the accessory device obtains the user device GUI from the video frames based on the metadata and presents the user device GUI on a display of the accessory device.
16. The system of claim 15 , wherein the instructions cause the processors to perform operations comprising:
receiving, by the user device, composite graphical user interface (GUI) configuration data from the accessory device, the composite GUI configuration data defining a maximum GUI display area and a plurality of view area definitions.
17. The system of claim 15 , wherein the instructions cause the processors to perform operations comprising:
generating, by the user device, an external display buffer having a size corresponding to the maximum GUI display area;
generating, by the user device, a user device GUI in a third portion of the external display buffer corresponding to the first portion of the maximum GUI display area; and
generating, by the user device, the video frames based on the external display buffer.
18. The system of claim 15 , wherein the instructions cause the processors to perform operations comprising:
receiving, by the user device, an update view area definition from the accessory device that identifies a fourth portion of the maximum GUI display area for generating the user device GUI; and
generating, by the user device, video frames having dimensions corresponding to the maximum GUI display area and including a user device GUI in a fifth portion of the video frames corresponding to the fourth portion of the maximum GUI display area.
19. The system of claim 18 , wherein the instructions cause the processors to perform operations comprising:
generating, by the user device, a transition animation for transitioning from the current view area to the updated view area by generating the user device GUI within intermediate view areas between the current view area and the updated view area within the video frames;
generating, by the user device, video frame metadata that describes a size and location of the intermediate view areas within the video frames; and
sending the video frames and the video frame metadata to the accessory device, where the accessory device uses the intermediate view areas and the metadata to generate a composite GUI that presents the transition animation.
20. The system of claim 18 , wherein the instructions cause the processors to perform operations comprising:
receiving, by the user device from the accessory device, user input related to the user device GUI presented by the accessory device;
determining, by the user device, a second view area based on the user input;
sending, by the user device, a request to transition to the second view area; and
receiving, by the user device, the updated view area definition that corresponds to the second view area.
21. The system of claim 15 , wherein the instructions cause the processors to perform operations comprising:
determining, by the user device, a safe area within the current view area based on the view area definition; and
determining, by the user device, at least one important feature of the user device GUI; and
generating, by the user device, the at least one important feature within the safe area of the current view area.