Automatic screen navigation for media device configuration and control
Methods, systems, and apparatuses are described for automatically navigating a graphical user interface (GUI) of a media device. In embodiments, objects (e.g. selectable items) in device GUI menu screens may be located, identified, and selected by another device (e.g., a switch) using several types of image processing, image recognition, and automated screen navigation techniques. For instance, a device may receive an operation request, for example from a user, receive a video signal from the connected media device, extract a video frame, process the video frame using one or more algorithms, and automatically navigate the GUI of the device by transmitting one more navigation commands for application to the GUI. In this manner, multi-step, time-consuming manual GUI navigation to achieve a desired result may be reduced to an automated technique, allowing users to save time and frustration in navigating a GUI using a remote control to find a particular resource.
1. A method of automatically controlling a media device, the method comprising:
obtaining, in a first device, configuration information to configure the media device; and
automatically interacting with a graphical user interface (GUI) provided by the media device to apply the configuration information to the media device, wherein the automatically interacting comprises performing the steps of:
receiving a video signal from the media device;
extracting one or more video frames from the video signal, each of the one or more video frames comprising a representation of the GUI of the media device;
processing the one or more video frames to obtain GUI information therefrom, the GUI information identifying a plurality of selectable objects presented in the GUI;
identifying a selection of a first selectable object from among the plurality of selectable objects based at least on the GUI information;
determining if the GUI is in a state to apply the configuration information to the media device based at least on the selection of the first selectable object;
in response to a determination that the GUI is not in a state to apply the configuration information, causing the GUI to be in the state to apply the configuration information by at least:
determining one or more navigation commands to cause a change in a state of the GUI from the selection of the first selectable object to a selection of a second selectable object; and
transmitting the one or more navigation commands to the media device based on the GUI information and the configuration information; and
in response to a determination that the GUI is in a state to apply the configuration information, transmitting a signal for application to the GUI to apply the configuration information.
2. The method of claim 1 , wherein applying the configuration information to the media device comprises downloading an application on the media device.
3. The method of claim 1 , wherein applying the configuration information to the media device comprises inputting user login information for an application.
4. The method of claim 1 , wherein applying the configuration information to the media device comprises modifying a user profile setting.
5. The method of claim 1 , wherein applying the configuration information to the media device comprises at least one of modifying a video setting, an audio setting, a power setting, or a High-Definition Multimedia Interface Consumer Electronics Control (HDMI-CEC) setting of the media device.
6. The method of claim 1 , wherein the obtaining the configuration information comprises obtaining the configuration information from a remote device over a network.
7. The method of claim 1 , wherein the determining if the GUI is in the state to apply the configuration information comprises determining whether an object corresponding to a configuration setting is selected in the GUI based on the GUI information.
8. A system for automatically controlling a media device, the system comprising:
a first device configured to obtain configuration information to configure the media device; and
one or more processing components configured to automatically interact with a graphical user interface (GUI) provided by the media device to apply the configuration information to the media device, the automatic interaction comprising:
receiving a video signal from the media device;
extracting one or more video frames from the video signal, each of the one or more video frames comprising a representation of the GUI of the media device;
processing the one or more video frames to obtain GUI information therefrom, the GUI information identifying a plurality of selectable objects presented in the GUI;
identifying a selection of a first selectable object from among the plurality of selectable objects based at least on the GUI information;
determining if the GUI is in a state to apply the configuration information to the media device based at least on the selection of the first selectable object;
in response to a determination that the GUI is not in a state to apply the configuration information, causing the GUI to be in the state to apply the configuration information by at least:
determining one or more navigation commands to cause a change in a state of the GUI from the selection of the first selectable object to a selection of a second selectable object; and
transmitting the one or more navigation commands to the media device based on the GUI information and the configuration information; and
in response to a determination that the GUI is in a state to apply the configuration information, transmitting a signal for application to the GUI to apply the configuration information.
9. The system of claim 8 , wherein the configuration information includes an instruction to download an application on the media device.
10. The system of claim 8 , wherein the configuration information includes user login information for an application.
11. The system of claim 8 , wherein the configuration information includes a modification of a user profile setting.
12. The system of claim 8 , wherein the configuration information includes at least one of a modification of a video setting, an audio setting, a power setting, or a High-Definition Multimedia Interface Consumer Electronics Control (HDMI-CEC) setting of the media device.
13. The system of claim 8 , wherein the first device is configured to obtain the configuration information from a remote device over a network.
14. The system of claim 8 , wherein the determining if the GUI is in the state to apply the configuration information comprises determining whether an object corresponding to a configuration setting is selected in the GUI based on the GUI information.
15. A multimedia switching device for configuring a media device, the multimedia switching device comprising:
one or more processing components configured to automatically interact with a graphical user interface (GUI) provided by the media device to apply configuration information to the media device, the automatic interaction comprising:
receiving a video signal from the media device;
extracting one or more video frames from the video signal, each of the one or more video frames comprising a representation of the GUI of the media device;
processing the one or more video frames to obtain GUI information therefrom, the GUI information identifying a plurality of selectable objects presented in the GUI;
identifying a selection of a first selectable object from among the plurality of selectable objects based at least on the GUI information;
determining if the GUI is in a state to apply the configuration information to the media device based at least on the selection of the first selectable object;
in response to a determination that the GUI is not in a state to apply the configuration information, causing the GUI to be in the state to apply the configuration information by at least:
determining one or more navigation commands to cause a change in a state of the GUI from the selection of the first selectable object to a selection of a second selectable object; and
transmitting the one or more navigation commands to the media device based on the GUI information and the configuration information; and
in response to a determination that the GUI is in a state to apply the configuration information, transmitting a signal for application to the GUI to apply the configuration information.
16. The multimedia switching device of claim 15 , wherein the configuration information includes an instruction to download an application on the media device.
17. The multimedia switching device of claim 15 , wherein the configuration information includes user login information for an application.
18. The multimedia switching device of claim 15 , wherein the configuration information includes a modification of a user profile setting.
19. The multimedia switching device of claim 15 , wherein the configuration information includes at least one of a modification of a video setting, an audio setting, a power setting, or a High-Definition Multimedia Interface Consumer Electronics Control (HDMI-CEC) setting of the media device.
20. The multimedia switching device of claim 15 , wherein the determining if the GUI is in the state to apply the configuration information comprises determining whether an object corresponding to a configuration setting is selected in the GUI based on the GUI information.