Automatic screen navigation for media device configuration and control
Methods, systems, and apparatuses are described for automatically navigating a graphical user interface (GUI) of a media device. In embodiments, objects (e.g. selectable items) in device GUI menu screens may be located, identified, and selected by another device (e.g., a switch) using several types of image processing, image recognition, and automated screen navigation techniques. For instance, a device may receive an operation request, for example from a user, receive a video signal from the connected media device, extract a video frame, process the video frame using one or more algorithms, and automatically navigate the GUI of the device by transmitting one more navigation commands for application to the GUI. In this manner, multi-step, time-consuming manual GUI navigation to achieve a desired result may be reduced to an automated technique, allowing users to save time and frustration in navigating a GUI using a remote control to find a particular resource.
1. A method of automatically identifying one or more resources available via at least one of a plurality of media devices, the method comprising:
for each of the plurality of media devices, automatically navigating a graphical user interface (GUI) of the media device, wherein the automatically navigating comprises:
receiving a video signal from the media device;
extracting one or more video frames from the video signal, each of the one or more video frames comprising a representation of the GUI of the media device;
processing the one or more video frames to obtain GUI information therefrom, the GUI information identifying one or more selectable objects presented in the GUI;
transmitting one or more navigation commands to the media device, based at least on the GUI information, to navigate to a GUI screen identifying applications executable on the media device;
determining at least one executable application available on the media device based at least on GUI information of the GUI screen; and
storing, in a first device, a mapping of the at least one executable application to the media device;
receiving an operation request to execute a particular application;
determining, based on the mapping, which media device of the plurality of media devices contains the particular application;
automatically navigating the GUI of the media device that contains the particular application; and
transmitting a command to the GUI of the media device that contains the particular application to execute the particular application.
2. The method of claim 1 , wherein the automatically navigating the GUI of the media device that contains the particular application comprises:
automatically navigating the GUI of the media device that contains the particular application until it is determined that an object selected on the GUI of the media device that contains the particular application corresponds to the operation request.
3. The method of claim 1 , further comprising displaying a list containing at least one of the executable applications on a sink device coupled to the first device.
4. The method of claim 1 , wherein the first device is a switching device, the switching device automatically selecting an input port corresponding to the media device that contains the particular application from the plurality of media devices to execute the particular application.
5. The method of claim 1 , wherein at least one of the executable applications in the mapping is an executable application stored on at least one of the plurality of media devices.
6. The method of claim 1 , wherein the mapping further includes a listing of multimedia content.
7. The method of claim 1 , wherein the automatically navigating the GUI of the media device for each of the plurality of media devices is performed while the first device is not transmitting the video signal to a sink device.
8. The method of claim 1 , wherein at least one of the plurality of media devices is a digital video recorder (DVR), and the mapping further comprises multimedia content recorded on the DVR.
9. A system for automatically identifying one or more resources available via at least one of a plurality of media devices, the system comprising:
one or more processing components; and
one or more memory devices that store program code including instructions to cause the one or more processing units to perform the actions of:
for each of the plurality of media devices, automatically navigating a graphical user interface (GUI) of the media device, the automatic navigation of the GUI including:
receiving a video signal from the media device;
extracting one or more video frames from the video signal, each of the one or more video frames comprising a representation of the GUI of the media device;
processing the one or more video frames to obtain GUI information therefrom, the GUI information identifying one or more selectable objects presented in the GUI;
transmitting one or more navigation commands to the media device, based at least on the GUI information, to navigate to a GUI screen identifying applications executable on the media device;
determining at least one executable application available on the media device based at least on GUI information of the GUI screen; and
storing, in a first device, a mapping of the at least one executable application to the media device;
receiving an operation request to execute a particular application;
determining, based on the mapping, which media device of the plurality of media devices contains the particular application;
automatically navigating the GUI of the media device that contains the particular application; and
transmitting a command to the GUI of the media device that contains the particular application to execute the particular application.
10. The system of claim 9 , wherein the automatically navigating the GUI of the media device that contains the particular application comprises:
automatically navigating the GUI of the media device that contains the particular application until it is determined that an object selected on the GUI of the media device that contains the particular application corresponds to the operation request.
11. The system of claim 9 , further comprising a display coupled to the first device for displaying a list containing at least one of the executable applications.
12. The system of claim 9 , wherein the first device is a switching device, the switching device configured to automatically select an input port corresponding to the media device that contains the particular application from the plurality of media devices to execute the particular application.
13. The system of claim 9 , wherein at least one of the executable applications in the mapping is an executable application stored on at least one of the plurality of media devices.
14. The system of claim 9 , wherein the mapping further includes a listing of multimedia content.
15. The system of claim 9 , wherein the automatically navigating the GUI of the media device for each of the plurality of media devices comprises automatically navigating the GUI while the first device does not transmit the video signal to a sink device.
16. The system of claim 9 , wherein at least one of the plurality of media devices is a digital video recorder (DVR), and the mapping further comprises multimedia content recorded on the DVR.
17. A multimedia switching device for automatically identifying one or more resources available via at least one of a plurality of media devices, the multimedia switching device comprising:
one or more processing components; and
one or more memory devices that store program code including instructions to cause the one or more processing units to perform the actions of:
for each of the plurality of media devices, automatically navigating a graphical user interface (GUI) of the media device, the automatic navigation of the GUI including:
receiving a video signal from the media device;
extracting one or more video frames from the video signal, each of the one or more video frames comprising a representation of the GUI of the media device;
processing the one or more video frames to obtain GUI information therefrom, the GUI information identifying one or more selectable objects presented in the GUI;
transmitting one or more navigation commands to the media device, based at least on the GUI information, to navigate to a GUI screen identifying applications executable on the media device;
determining at least one executable application available on the media device based at least on GUI information of the GUI screen; and
storing, in a first device, a mapping of the at least one executable application to the media device;
receiving an operation request to execute a particular application;
determining, based on the mapping, which media device of the plurality of media devices contains the particular application;
automatically navigating the GUI of the media device that contains the particular application; and
transmitting a command to the GUI of the media device that contains the particular application to execute the particular application.
18. The multimedia switching device of claim 17 , wherein the program code further includes instructions to perform the actions of causing a transmitter to transmit one or more navigation commands for application to the GUI of the media device to obtain additional information for at least one resource in the mapping.
19. The multimedia switching device of claim 17 , wherein the program code further includes instructions to perform the actions of:
automatically navigating the GUI of the media device that contains the particular application until it is determined that an object selected on the GUI of the media device that contains the particular application corresponds to the operation request.
20. The multimedia switching device of claim 17 , wherein at least one of the executable applications in the mapping is an executable application stored on at least one of the plurality of media devices.