Electronic apparatus and method for controlling the electronic apparatus thereof

1. An electronic apparatus, comprising:

at least one memory configured to store at least one instruction; and

at least one processor configured to execute the at least one instruction to perform a process including:

receiving, from an external device, data corresponding to a user interface (UI) for controlling the external device,

providing the received data to a trained model to obtain, from the received data, information on structure and location of a UI element in the UI,

based on an image being received by the electronic apparatus from the external device:

identifying whether a UI is included in the received image,

based on identifying that a UI is included in the received image:

comparing information on structure and location of a UI element in the UI included in the received image with the information on the structure and the location of the UI element in the UI obtained from the received data,

identifying, based on the compared information, a similarity value between the information on the structure and the location of the UI element in the UI included in the received image and the information on the structure and the location of the UI element in the UI obtained from the received data,

identifying, based on the identified similarity value being equal to or more than a predetermined value, that there is not a change from the UI to which the received data corresponds to the UI included in the received image,

identifying, based on the identified similarity value being less than the predetermined value, that there is a change from the UI to which the received data corresponds to the UI included in the received image, and

based on identifying that there is a change from the UI to which the received data corresponds to the UI included in the received image:

retraining the trained model for obtaining the information on the structure and the location of the UI element in the UI obtained from the received data,

storing the information on the structure and the location of the UI element in the UI obtained from the received data in the at least one memory,

matching a function of the UI included in the received image with a control button of a remote controller for controlling the electronic apparatus, and

storing a result of the matching in the at least one memory.

2. The electronic apparatus as claimed in claim 1 , wherein

the providing the received data to the trained model provides the received data to the trained model to obtain information on a logo included in the UI to which the received data corresponds, and

the process further includes, based on identifying that the UI is included in the received image, comparing information on a logo included in the UI included in the received image and the information on the logo included in the UI to which the received data corresponds.

3. The electronic apparatus as claimed in claim 2 , wherein

the comparing information on the logo included in the UI included in the received image and the information on the logo included in the UI to which the received data corresponds compares the information on the logo included in the UI included in the received image and the information on the logo included in the UI to which the received data corresponds, to identify whether a provider that provides a content via the external device has changed.

4. The electronic apparatus as claimed in claim 3 , wherein the process further comprises, based on identifying that a provider that provides a content via the external device has changed:

providing a guide message that the provider has changed.

5. The electronic apparatus as claimed in claim 1 , wherein

the receiving, from the external device, the data corresponding to the UI for controlling the external device includes:

receiving, from the external device, an image including the UI for controlling the external device, and

obtaining the UI for controlling the external device from the received image.

6. The electronic apparatus as claimed in claim 1 , wherein

the process further comprises, based on identifying that a UI is included in the received image:

obtaining the information on the location of the UI element in the UI included in the received image by identifying an area in the received image in which a content is displayed and an area in the received image in which a UI is displayed, and

obtaining the information on the structure of the UI element in the UI included in the received image by extracting a feature area of the UI included in the received image.

7. The electronic apparatus as claimed in claim 1 , further comprising:

a communication interface for communicating with the remote controller for controlling the electronic apparatus; and

a user interface,

wherein the process further comprises, in accordance with a user command being input from the remote controller through the communication interface, or being input through the user interface:

providing, to the external device, a request for an image according to the user command, and

in accordance with an image being received from the external device based on the request, identify whether a UI is included in the image received based on the request, by identifying an area in the image received based on the request in which a content is displayed and an area in the image received based on the request in which a UI is displayed.

8. The electronic apparatus as claimed in claim 1 , wherein the process further comprises:

receiving, from the external device, information on the external device,

storing the received information on the external device in the at least one memory, and

after storing the received information on the external device, and after the external device has been connected again to the electronic apparatus after being disconnected from the electronic apparatus:

receiving information on the external device connected again to the electronic apparatus, and

identifying whether a provider that provides a content to the electronic apparatus through the external device has changed by comparing the information on the external device stored in the at least one memory with the received information on the external device connected again.

9. The electronic apparatus as claimed in claim 1 ,

wherein the information on the structure of the UI element in the UI to which the received data corresponds includes information on size and shape of the UI element to which the received data corresponds, and

the information on the structure of the UI element in the UI included in the received image includes information on size and shape of the UI element in the UI included in the received image.

10. A method comprising, by an electronic apparatus:

receiving, from an external device, data corresponding to a user interface (UI) for controlling the external device;

providing the received data to a trained model to obtain, from the received data, information on structure and location of a UI element in the UI;

based on an image being received by the electronic apparatus from the external device:

identifying whether a UI is included in the received image,

based on identifying that a UI is included in the received image:

comparing information on structure and location of a UI element in the UI included in the received image with the information on the structure and the location of the UI element in the UI obtained from the received data,

identifying, based on the compared information, a similarity value between the information on the structure and the location of the UI element in the UI included in the received image and the information on the structure and the location of the UI element in the UI obtained from the received data,

identifying, based on the identified similarity value being equal to or more than a predetermined value, that there is not a change from the UI to which the received data corresponds to the UI included in the received image, and

identifying, based on the identified similarity value being less than the predetermined value, that there is a change from the UI to which the received data corresponds to the UI included in the received image, and

based on identifying that there is a change from the UI to which the received data corresponds to the UI included in the received image:

retraining the trained model for obtaining the information on the structure and the location of the UI element in the UI obtained from the received data,

storing the information on the structure and the location of the UI element in the UI obtained from the received data in a memory,

matching a function of the UI included in the received image with a control button of a remote controller for controlling the electronic apparatus, and

storing a result of the matching in the memory.

11. The method as claimed in claim 10 , wherein

the providing the received data to a trained model provides the received data to the trained model to obtain information on a logo included in the UI to which the received data corresponds, and

the method further comprises, based on identifying that a UI is included in the received image, comparing information on a logo included in the UI included in the received image and the information on the logo included in the UI to which the received data corresponds.

12. The method as claimed in claim 11 , wherein

the receiving, from the external device, the data corresponding to the UI for controlling the external device includes:

receiving, from the external device, an image including the UI for controlling the external device, and

obtaining the UI for controlling the external device from the received image.

13. The method as claimed in claim 11 , wherein

the method further comprises, based on identifying that a UI is included in the received image:

obtaining the information on the location of the UI element in the UI included in the received image by identifying an area in the received image in which a content is displayed and an area in the received image in which a UI is displayed, and

obtaining the information on the structure of the UI element in the UI included in the received image by extracting a feature area of the UI included in the received image.

14. The method as claimed in claim 11 , wherein

the comparing the information on the logo included in the UI included in the received image and the information on the logo included in the UI to which the received data corresponds compares the information on the logo included in the UI included in the received image and the information on the logo included in the UI to which the received data corresponds, to identify whether a provider that provides a content via the external device has changed.

15. The method as claimed in claim 14 , further comprising, based on identifying that a provider that provides a content via the external device has changed:

providing a guide message that the provider has changed.

16. The method as claimed in claim 10 , further comprising:

receiving a user command input to the electronic apparatus from the remote controller for controlling the electronic apparatus or input to the electronic apparatus through a user interface of the electronic apparatus;

providing, to the external device, a request for an image according to the received user command, and

in accordance with an image being received from the external device based on the request, identifying an area in the image received based on the request in which a content is displayed and an area in the image received based on the request in which a UI is displayed.

17. The method as claimed in claim 10 , further comprising:

receiving, from the external device, information on the external device;

storing the received information on the external device in the memory; and

after storing the received information on the external device, and after the external device has been connected again to the electronic apparatus after being disconnected from the electronic apparatus,

receiving information on the external device connected again to the electronic apparatus, and

identifying whether a provider that provides a content to the electronic apparatus through the external device has changed by comparing the information on the external device stored in the memory with the received information on the external device connected again.

18. An electronic apparatus, comprising:

at least one memory configured to store at least one instruction; and

at least one processor configured to execute the at least one instruction to perform a process including:

receiving, from an external device, data corresponding to a user interface (UI) for controlling the external device,

obtaining, from the received data, information on structure and location of a UI element in the UI,

based on an image being received by the electronic apparatus from the external device:

identifying whether a UI is included in the received image,

based on identifying that a UI is included in the received image:

comparing information on structure and location of a UI element in the received image with the information on the structure and the location of the UI element in the UI obtained from the received data

identifying, based on the compared information, a similarity value between the information on the structure and the location of the UI element in the UI included in the received image and the information on the structure and the location of the UI element in the UI obtained from the received data,

identifying, based on the identified similarity value being equal to or more than a predetermined value, that there is not a change from the UI to which the received data corresponds to the UI included in the received image, and

identifying, based on the identified similarity value being less than the predetermined value, that there is a change from the UI to which the received data corresponds to the UI included in the received image, and

based on identifying that there is a change from the UI to which the received data corresponds to the UI included in the received image:

retraining a trained model for obtaining the information on the structure and the location of the UI element in the UI obtained from the received data,

storing the information on the structure and the location of the UI element in the UI obtained from the received data in the at least one memory,

matching a function of the UI included in the received image with a control button of a remote controller for controlling the electronic apparatus, and

storing a result of the matching in the at least one memory.