Systems and methods of multimodal user interface automation

1 . A method comprising:

storing, on a computing device, an objective for automated manipulation of a user interface (UI) of a user device;

conducting, by the computing device, an iterative sequence on the UI to achieve the objective, wherein the iterative sequence includes:

capturing current state information of the UI, wherein the current state information includes text and an image of the UI,

detecting changes in the current state information from stored previous state information,

selecting, from a component hierarchy of code for the UI, relevant sections related to the changes in the current state information,

sending multimodal input to a generative artificial intelligence (AI) system, wherein the multimodal input includes the image of the UI, a text prompt, the objective, and the relevant sections,

receiving, from the generative AI system, a predicted next action for the UI based on the multimodal input,

initiating a programmatic action to implement the predicted next action on the UI,

recording the programmatic action; and

generating, by the computing device and based on the iterative sequence, a UI automation model associated with achieving the objective.

2 . The method of claim 1 , wherein capturing includes obtaining the image of the UI and the component hierarchy.

3 . The method of claim 1 , wherein the UI includes one of:

a graphical user interface (GUI) for an application executed on the user device,

a GUI for a web browser executed on the user device, or

a UI of an application that is embedded in another application.

4 . The method of claim 1 , further comprising:

storing, on the computing device, rules governing actions related to the objective.

5 . The method of claim 4 , wherein the iterative sequence further includes:

selecting, from the rules, relevant rules related to the changes in the current state information, wherein the multimodal input further includes the relevant rules.

6 . The method of claim 1 , wherein the iterative sequence further includes:

performing prompt engineering to shorten the component hierarchy for the text prompt.

7 . The method of claim 1 , wherein initiating a programmatic action includes simulating a physical user action on the UI.

8 . The method of claim 1 , wherein the UI automation model includes a sequence of programmatic actions for achieving the objective.

9 . A device comprising:

a communication interface; and

one or more processors configured to:

store in a memory an objective for automated manipulation of a user interface (UI) of a user device;

conduct an iterative sequence to achieve the objective, wherein the iterative sequence includes:

capturing current state information of the UI, wherein the current state information includes text and an image of the UI,

detecting changes in the current state information from stored previous state information,

selecting, from a component hierarchy of code for the UI, relevant sections related to the changes in the current state information,

sending multimodal input to a generative artificial intelligence (AI) system, wherein the multimodal input includes the image of the UI, a text prompt, the objective, and the relevant sections,

receiving, from the generative AI system, a predicted next action for the UI based on the multimodal input,

initiating a programmatic action to implement the predicted next action on the UI, and

recording the programmatic action; and

generating, based on the iterative sequence, a UI automation model associated with achieving the objective.

10 . The device of claim 9 , wherein capturing includes obtaining the image of the UI and the component hierarchy.

11 . The device of claim 9 , wherein the UI includes one of:

a graphical user interface (GUI) for an application executed on the user device,

a GUI for a web browser executed on the user device, or

a UI of an application that is embedded in another application.

12 . The device of claim 9 , wherein the one or more processors are further configured to:

store, in the memory, rules governing actions related to the objective.

13 . The device of claim 12 , wherein, when conducting the iterative sequence, the one or more processors are further configured to:

select, from the rules, relevant rules related to the current state information, wherein the multimodal input further includes the relevant rules.

14 . The device of claim 12 , wherein, when conducting the iterative sequence, the one or more processors are further configured to:

select, from the rules, relevant rules related to the changes in the current state information, wherein the multimodal input further includes the relevant rules.

15 . The device of claim 9 , wherein, when conducting the iterative sequence, the one or more processors are further configured to:

shorten the component hierarchy prior to sending the multimodal input.

16 . The device of claim 9 , wherein, when initiating the programmatic action, the one or more processors are further configured to:

simulate a physical user action on the UI.

17 . The device of claim 9 , wherein the UI automation model includes a sequence of programmatic actions for achieving the objective.

18 . A non-transitory, computer-readable storage medium storing instructions executable by a processor of a computing device for:

storing, in a memory, an objective for automated manipulation of a user interface (UI) of a user device;

conducting, by the computing device, an iterative sequence to achieve the objective, wherein the iterative sequence includes:

capturing current state information of the UI, wherein the current state information includes text and an image of the UI,

detecting changes in the current state information from stored previous state information,

selecting, from a component hierarchy of code for the UI, relevant sections related to the changes in the current state information,

sending multimodal input to a generative artificial intelligence (AI) system, wherein the multimodal input includes the image of the UI, a text prompt, the objective, and the relevant sections,

receiving, from the generative AI system, a predicted next action for the UI based on the multimodal input,

initiating a programmatic action to implement the predicted next action on the UI, and

recording the programmatic action; and

generating, by the computing device and based on the iterative sequence, a UI automation model associated with achieving the objective.

19 . The non-transitory, computer-readable storage medium of claim 18 , further storing instructions executable by the processor of a computing device for:

storing rules governing actions related to the objective, and

sending, to the generative AI system, the rules.

20 . The non-transitory, computer-readable storage medium of claim 18 , wherein the instructions for conducting the iterative sequence further include instructions executable by the processor of a computing device for:

repeating the iterative sequence, if the objective is not achieved after initiating the programmatic action.