Granted Patent B2

US 12,681,629 · App. 18/191,862 · Granted Jul 14, 2026

Inventors: Chun Hong Zheng (Beijing, CN); Qun Q Zhang (Shanghai, CN); Xiao Feng Ji (Shanghai, CN)

Assignee: International Business Machines Corporation

G06F3/04847G06N3/045G06N3/0475G06N3/094

Abstract

Claims (73)

1 . A method, comprising:

determining, using a processor, an execution-step sequence for a first graphical user interface, wherein determining the execution-step sequence comprises:

determining, by an encoder of a Generative Adversarial Network (GAN) and using one or more components of the first graphical user interface, a hidden state associated with the one or more components; and

determining, by a generator of the GAN and using the hidden state and a target function, the execution-step sequence;

presenting a second graphical user interface, the second graphical user interface comprising:

a first panel in which the first graphical user interface is presented;

a second panel comprising a first user selectable component via which a user navigates forward in the execution-step sequence for the first graphical user interface and a second user selectable component via which the user navigates backward in the execution-step sequence for the first graphical user interface; and

a third panel configured to represent a plurality of execution-steps of the execution-step sequence that are executed, wherein the first panel, the second panel, and the third panel are separate and distinct; and

responsive to the user selecting the first user selectable component:

executing an execution-step in the execution-step sequence;

visually depicting a result of the executing the execution-step by redrawing the first graphical user interface; and

adding component status data to the third panel, wherein the component status data indicates, for the execution-step:

an action associated with the execution-step; and

a particular component of the one or more components on which the action was performed.

2 . The method of claim 1 , wherein the redrawing the first graphical user interface comprises:

redrawing a graphical user interface component, of the first graphical user interface, pertaining to the execution-step in the execution-step sequence that was executed.

3 . The method of claim 2 , wherein the redrawing the graphical user interface component comprises redrawing the graphical user interface component using Component Status Transition Generative Adversarial Networks trained using machine learning.

4 . The method of claim 1 , further comprising:

responsive to the user selecting the second user selectable component, removing from the third panel the component status data indicating the execution-step in the execution-step sequence that was executed responsive to the user selecting the first user selectable component.

5 . The method of claim 1 , wherein the determining the execution-step sequence for the first graphical user interface comprises determining the execution-step sequence using Function Execution-step Generative Adversarial Networks trained using machine learning.

6 . The method of claim 1 , further comprising:

generating a component status table for each execution-step in the execution-step sequence, each component status table tracking status of graphical user interface components of the first graphical user interface at the respective execution-step.

7 . The method of claim 1 , wherein:

the second panel comprises a third user selectable component; and

responsive to the user selecting the third user selectable component, permitting the user to tune the execution-steps of the execution-step sequence.

8 . The method of claim 1 , further comprising:

outputting content comprising a plurality of execution-steps of the execution-step sequence and, for each of the plurality of execution-steps, a respective image of the first graphical user interface corresponding to that execution-step and depicting the first graphical user interface in a state resulting from that execution-step.

9 . A system, comprising:

a processor programmed to initiate executable operations comprising:

determining an execution-step sequence for a first graphical user interface, wherein determining the execution-step sequence comprises:

determining, by an encoder of a Generative Adversarial Network (GAN) and using one or more components of the first graphical user interface, a hidden state associated with the one or more components; and

determining, by a generator of the GAN and using the hidden state and a target function, the execution-step sequence;

presenting a second graphical user interface, the second graphical user interface comprising:

a first panel in which the first graphical user interface is presented;

a second panel comprising a first user selectable component via which a user navigates forward in the execution-step sequence for the first graphical user interface and a second user selectable component via which the user navigates backward in the execution-step sequence for the first graphical user interface; and

a third panel configured to represent steps of the execution-step sequence that are executed, wherein the first panel, the second panel, and the third panel are separate and distinct; and

responsive to the user selecting the first user selectable component:

executing an execution-step in the execution-step sequence;

visually depicting a result of the executing the execution-step by redrawing the first graphical user interface; and

adding component status data to the third panel, wherein the component status data indicates, for the execution-step:

an action associated with the execution-step; and

a particular component of the one or more components on which the action was performed.

10 . The system of claim 9 , wherein the redrawing the first graphical user interface comprises:

redrawing a graphical user interface component, of the first graphical user interface, pertaining to the execution-step in the execution-step sequence that was executed.

11 . The system of claim 10 , wherein the redrawing the graphical user interface component comprises redrawing the graphical user interface component using Component Status Transition Generative Adversarial Networks trained using machine learning.

12 . The system of claim 9 , the executable operations further comprising:

responsive to the user selecting the second user selectable component, removing from the third panel the component status data indicating the execution-step in the execution-step sequence that was executed responsive to the user selecting the first user selectable component.

13 . The system of claim 9 , wherein the determining the execution-step sequence for the first graphical user interface comprises determining the execution-step sequence using Function Execution-step Generative Adversarial Networks trained using machine learning.

14 . The system of claim 9 , the executable operations further comprising: generating a component status table for each execution-step in the execution-step sequence, each component status table tracking status of graphical user interface components of the first graphical user interface at the respective execution-step.

15 . The system of claim 9 , wherein:

the second panel comprises a third user selectable component; and

responsive to the user selecting the third user selectable component, permitting the user to tune the execution-steps of the execution-step sequence.

16 . The system of claim 9 , the executable operations further comprising: outputting content comprising a plurality of execution-steps of the execution-step sequence and, for each of the plurality of execution-steps, a respective image of the first graphical user interface corresponding to that execution-step and depicting the first graphical user interface in a state resulting from that execution-step.

17 . A computer program product, comprising:

one or more computer readable storage mediums having program code stored thereon, the program code stored on the one or more computer readable storage mediums collectively executable by a data processing system to initiate operations including:

determining an execution-step sequence for a first graphical user interface, wherein determining the execution-step sequence comprises:

determining, by an encoder of a Generative Adversarial Network (GAN) and using one or more components of the first graphical user interface, a hidden state associated with the one or more components; and

determining, by a generator of the GAN and using the hidden state and a target function, the execution-step sequence;

presenting a second graphical user interface, the second graphical user interface comprising:

a first panel in which the first graphical user interface is presented;

a second panel comprising a first user selectable component via which a user navigates forward in the execution-step sequence for the first graphical user interface and a second user selectable component via which the user navigates backward in the execution-step sequence for the first graphical user interface; and

a third panel configured to represent steps of the execution-step sequence that are executed, wherein the first panel, the second panel, and the third panel are separate and distinct; and

responsive to the user selecting the first user selectable component:

executing an execution-step in the execution-step sequence;

visually depicting a result of the executing the execution-step by redrawing the first graphical user interface; and

adding component status data to the third panel, wherein the component status data indicates, for the execution-step:

an action associated with the execution-step; and

a particular component of the one or more components on which the action was performed.

18 . The computer program product of claim 17 , wherein the redrawing the first graphical user interface comprises:

redrawing a graphical user interface component, of the first graphical user interface, pertaining to the execution-step in the execution-step sequence that was executed.

19 . The computer program product of claim 18 , wherein the redrawing the graphical user interface component comprises redrawing the graphical user interface component using Component Status Transition Generative Adversarial Networks trained using machine learning.

20 . The computer program product of claim 17 , wherein the program code is executable by the data processing system to initiate operations further comprising:

responsive to the user selecting the second user selectable component, removing from the third panel the component status data indicating the execution-step in the execution-step sequence that was executed responsive to the user selecting the first user selectable component.

Assignments (1)

Continuity (1)

References Cited (27)