Method, non-transitory computer-readable medium, and device for controlling a user-interface

1. A method for controlling a user-interface that provides an instruction to an application through a user-operation which is performed on a display provided on a device, the method comprising a process performed by the device, wherein the process includes:

acquiring information displayed by the display;

extracting at least one feature existing on the acquired information;

receiving an action of a user;

searching a database to identify a predetermined operation that corresponds to the received action and the extracted at least one feature, and to identify, even for the same received action, a different predetermined operation for a different extracted at least one feature; and

providing an instruction to the application through applying the identified predetermined operation to the user-interface.

2. The method according to claim 1 , wherein the extracting extracts the at least one feature, by means of pattern recognition of the display.

3. The method according to claim 1 , wherein the action is a physical motion of the user that is captured by a camera or a voice of the user collected by a microphone.

4. The method according to claim 1 , wherein the application runs on another device that is different from the device provided with the screen.

5. A non-transitory computer-readable medium storing a program for causing the device provided with the screen to execute the method according to claim 1 .

6. A device for controlling a user-interface that provides an instruction to an application through a user-operation which is performed on a display provided on the device, wherein the device comprising a memory having computer readable instructions; and

at least one processor configured to execute the computer readable instructions, to

acquire information displayed by the display;

extract at least one feature existing on the acquired information;

receive an action of a user;

search a database to identify a predetermined operation that corresponds to the received action and the extracted at least one feature, and to identify, even for the same received action, a different predetermined operation for a different extracted at least one feature; and

provide an instruction to the application through applying the identified predetermined operation to the user-interface.

7. The method according to claim 1 , wherein

the predetermined operation to be identified is a first operation for a case where the extracted at least on feature is a first feature, and the predetermined operation to be identified is a second operation different from the first operation for a case where the extracted at least one feature is a second feature different from the first feature.

8. The device according to claim 6 , wherein

the predetermined operation to be identified is a first operation for a case where the extracted at least one feature is a first feature, and the predetermined operation to be identified is a second operation different from the first operation for a case where the extracted at least one feature is a second feature different from the first feature.