Systems and Methods for Configuring Artificial Intelligence Agents to Automate Multimodal Interface Workflows

1 . A system for configuring an agent to automate multimodal interface workflows, comprising:

a memory;

one or more processors coupled to the memory and configured to:

construct agent specification functions using prompts and agent functions, wherein the agent specifications are configured to automate a multimodal interface workflow, and wherein the agent specification functions are configured to construct multiple levels of prompts including click-level prompts, step-level prompts, and task-level prompts, and wherein the agent specification functions conceal complexities of a webpage document object model (DOM) by abstracting interface elements into higher-level references; and

generate multimodal agent calls, by translating the agent specification functions, that cause an agent to implement the agent functions to produce outputs that performs tasks corresponding to the prompts, wherein the agent calls triggers one or more machine-actuated actions based on the outputs as synthetic actions that automate the multimodal interface workflow, wherein automating the multimodal interface workflow includes modifying elements of the interface, and wherein the interface is a user interface.

2 . The system of claim 1 , wherein the agent specification functions are constructable using various degrees of expressiveness.

3 . The system of claim 2 , wherein the agent specification functions are constructed using natural language commands.

4 . The system of claim 3 , wherein the agent specification functions are constructed using prescriptive commands.

5 . The system of claim 4 , wherein the agent specification functions are constructed using a combination of the natural language commands and the prescriptive commands.

6 . (canceled)

7 . The system of claim 2 , wherein a first agent specification function include interface operations and interface elements.

8 . The system of claim 7 , wherein a second agent specification function do not include the interface operations and the interface elements.

9 . The system of claim 8 , wherein the agent functions apply the interface operations to the interface elements.

10 . The system of claim 1 , wherein the outputs are a sequence of actuation commands.

11 . (canceled)

12 . The system of claim 11 , further comprises an actuator configured to receive the sequence of actuation commands from the agent, and process the actuation commands to trigger one or more machine-actuated actions based on the sequence of actuation commands as synthetic actions that automate the multimodal interface workflow.

13 . (canceled)

14 . The system of claim 1 , wherein the agent functions include answerQuestionAboutScreen, goToURL, act, typeIntoElement, click, type, wait, goToSong, compose, answerTrueFalseQuestionAboutScreen, composeAndType, getCurrentDate, pickdate, fillform, isVisible, keydown, print, scroll, and spotlight.

15 . The system of claim 1 , wherein the interface operations include clicks, hovers, scrolls, picks, text entries, and form fills.

16 . The system of claim 1 , wherein the processor is further configured to construct the agent specification functions using another agent.

17 . The system of claim 16 , wherein the another agent is a large language model (LLM).

18 . The system of claim 16 , wherein the processor is further configured to receive a preliminary agent specification from the another agent, and to receive edits from a user to the preliminary agent specification functions to generate a final agent specification.

19 . The system of claim 16 , wherein the processor is further configured to construct the agent specification functions using atomic actions.

20 . The system of claim 19 , wherein the atomic actions include CLICK, TYPE, SCROLL, and ACT.