Method and electronic device for navigating application screen

1. A method, performed by an electronic device, of navigating a screen of an application, the method comprising:

receiving a user input;

determining, based on the user input, a user intent for controlling the electronic device;

determining a command for performing a control operation corresponding to the user intent as a goal;

generating layout information by identifying functions of user interface (UI) elements, wherein a UI element causes a corresponding function to perform when selected by the user, and the layout information includes positions, sizes, and the functions of the UI elements;

determining, based on the user intent and the layout information, at least one sub-goal for executing the command, wherein the determining of the at least one sub-goal comprises:

detecting a current state of the electronic device based on the layout information;

determining, based on the user intent, a target state of the electronic device in which the command is executed; and

determining the at least one sub-goal based on the current state of the electronic device and the target state of the electronic device;

executing the command by performing at least one task corresponding to the at least one sub-goal based on the at least one sub-goal including two or more sub-goals, sequentially achieving the two or more sub-goals until the command is executed;

performing the validation based on the screen of the application being switched from a first screen to a second screen, by identifying the UI elements in the second screen to determine whether a first sub-goal is achieved; and

changing a second sub-goal, which has been determined previously, based on whether the first sub-goal is achieved,

wherein the executing of the command includes navigating the application by performing the at least one task including switching of the screen of the application,

wherein the at least one sub-goal is changeable based on a validation of an operation of navigating the application for executing the command, and

the at least one task includes units of action for navigating the application.

2. The method of claim 1 , wherein the elements of the user interface of the screen of the application include at least one of an icon, an image, a text, or a button.

3. The method of claim 1 , further comprising determining whether the command corresponds to a function that is callable by an operating system of the electronic device,

wherein the generating layout information includes generating the layout information by analyzing the screen layout of the application based on the command corresponding to a function that is uncallable by the operating system of the electronic device.

4. The method of claim 1 , wherein the dynamically determining of the second sub-goal includes changing the second sub-goal based on layout information of the second screen.

5. The method of claim 4 , wherein the executing of the command includes:

performing a first task corresponding to the first sub-goal; and

performing a second task corresponding to the second sub-goal.

6. The method of claim 1 , wherein

the user input includes a voice signal, and

the determining of the user intent includes:

converting the voice signal into a text by performing automatic speech recognition (ASR) on the voice signal; and

obtaining the user intent by applying the text to a natural-language understanding (NLU) model.

7. An electronic device for navigating a screen of an application, the electronic device comprising:

a display configured to display the screen of the application being executed;

a communication interface;

a memory storing one or more instructions; and

a processor configured to execute the one or more instructions stored in the memory to:

receive a user input,

determine, based on the user input, a user intent for controlling the electronic device,

determine a command for performing a control operation corresponding to the user intent as a goal,

generate layout information by identifying functions of user interface (UI) elements, wherein a UI element causes a corresponding function to perform when selected by the user, and the layout information includes positions, sizes, and the functions of the UI elements,

determine, based on the user intent and the layout information, at least one sub-goal for executing the command, wherein the determining the at least one sub-goal comprises:

detect a current state of the electronic device based on the layout information,

determine, based on the user intent, a target state of the electronic device in which the operation is executed, and

determine the at least one sub-goal based on the current state of the electronic device and the target state of the electronic device,

execute the command by performing at least one task corresponding to the at least one sub-goal based on the at least one sub-goal including two or more sub-goals, sequentially achieve the two or more sub-goals until the control command is executed,

perform the validation based on the screen of the application being switched from a first screen to a second screen, by identifying the UI elements in the second screen to determine whether a first sub-goal is achieved, and

change a second sub-goal, which has been determined previously, based on whether the first sub-goal is achieved,

wherein the executing of the command includes navigating the application by performing the at least one task including switching of the screen of the application,

wherein the at least one sub-goal is changeable based on a validation on an operation of navigating the application for executing the command, and

the at least one task includes units of action for navigating the application.

8. The electronic device of claim 7 , wherein the processor is further configured to execute the one or more instructions to:

determine whether the command corresponds to a function that is callable by an operating system of the electronic device, and

generating the layout information by analyzing the screen layout of the application based on the command corresponding to a function that is uncallable by the operating system of the electronic device.

9. The electronic device of claim 7 , wherein the processor is further configured to execute the one or more instructions to change the second sub-goal based on layout information of the second screen.

10. The electronic device of claim 9 , wherein the processor is further configured to execute the one or more instructions to perform a first task corresponding to the first sub-goal, and perform a second task corresponding to the second sub-goal.

11. The electronic device of claim 7 , wherein

the user input includes a voice signal, and

the processor is further configured to execute the one or more instructions to convert the voice signal into a text by performing automatic speech recognition (ASR) on the voice signal, and obtain the user intent by applying the text to a natural-language understanding (NLU) model.

12. A non-transitory computer-readable recording medium having recorded thereon a program for navigating a screen of an application by executing a method comprising:

receiving a user input;

determining, based on the user input, a user intent for controlling the electronic device;

determining a command for performing a control operation corresponding to the user intent as a goal;

generating layout information by identifying functions of user interface (UI) elements, wherein a UI element causes a corresponding function to perform when selected by the user, and the layout information includes positions, sizes, and the functions of the UI elements;

determining, based on the user intent and the layout information, at least one sub-goal for executing the command, wherein the determining of the at least one sub-goal comprises:

detecting a current state of the electronic device based on the layout information;

determining, based on the user intent, a target state of the electronic device in which the command is executed; and

determining the at least one sub-goal based on the current state of the electronic device and the target state of the electronic device;

executing the command by performing at least one task corresponding to the at least one sub-goal based on the at least one sub-goal including two or more sub-goals, sequentially achieving the two or more sub-goals until the command is executed;

perform the validation based on the screen of the application being switched from a first screen to a second screen, by identifying the UI elements in the second screen to determine whether a first sub-goal is achieved; and

change a second sub-goal, which has been determined previously, based on whether the first sub-goal is achieved,

wherein the executing of the command includes navigating the application by performing the at least one task including switching of the screen of the application,

wherein the at least one sub-goal is changeable based on a validation of an operation of navigating the application for executing the command, and

the at least one task includes units of action for navigating the application.