Granted Patent B1

US 12,430,150 · App. 18/909,455 · Granted Sep 30, 2025

Inventors: Rohan Bavishi (San Francisco, CA); Lina Lukyantseva (San Francisco, CA); Shaya Zarkesh (San Francisco, CA); Kadhir Manickam (San Francisco, CA); Jacob van Gogh (San Francisco, CA); Frederick Robinson (San Francisco, CA); Rick Liu (San Francisco, CA); Vibhaa Sivaraman (San Francisco, CA); Matthew Elkherj (San Francisco, CA); Billy Wang (San Francisco, CA); Armaan Goel (San Francisco, CA); Bryan Schmidt (San Francisco, CA); Erich Elsen (San Francisco, CA); Curtis Hawthorne (San Francisco, CA)

Assignee: Anthropic, PBC

G06F9/451

Abstract

Claims (39)

1. A system, running on one or more processors, for client-side implementation of an interface automation language at runtime, comprising:

agent specification logic, running on client-side, and configured to construct an agent specification, and to make the agent specification available for server-side translation into an intermediate representation, wherein the agent specification is configured to automate a multimodal interface workflow; and

runtime interpretation logic, running on the client-side, and configured to:

receive the intermediate representation;

detect one or more agent functions in the intermediate representation;

generate one or more agent calls based on the agent functions;

issue the agent calls to an agent, and, in response, receive at least one runtime actuation function from the agent; and

translate the runtime actuation function into at least one runtime actuation command, wherein the runtime actuation command triggers at least one machine-actuated action as a runtime synthetic action that automates the multimodal interface workflow.

2. The system of claim 1 , wherein the agent functions include built-in functions, planner functions, and workflow functions.

3. The system of claim 2 , wherein the built-in functions include answerQuestionAboutScreen, goToURL, typeIntoElement, click, type, wait, goToSong, compose, answerTrueFalseQuestionAboutScreen, composeAndType, getCurrentDate, isVisible, keydown, print, scroll, and spotlight.

4. The system of claim 2 , wherein the planner functions include act, fillform, and pickdate.

5. The system of claim 4 , wherein the runtime interpretation logic is further configured to invoke an observation logic in response to detecting the act planner function.

6. The system of claim 5 , wherein the observation logic is configured to send one or more interface screenshots, an action history, and a task description to the agent.

7. The system of claim 6 , wherein the interface screenshots include a current interface screenshot and one or more previous interface screenshots.

8. The system of claim 6 , wherein the action history includes a current runtime actuation command and one or more previous runtime actuation commands.

9. The system of claim 6 , wherein the task description includes a description of the multimodal interface workflow.

10. The system of claim 1 , further comprising a prompt rendering logic that is configured to provide a system prompt, the interface screenshots, the action history, and the task description as model messages to the agent.

11. The system of claim 1 , further comprising a prompt rendering logic that is configured to provide a system prompt, the interface screenshots, the action history, and the task description as runtime agent messages to the agent.

12. The system of claim 1 , wherein the runtime interpretation logic is further configured to receive a return value from the agent in response to the agent calls.

13. The system of claim 12 , wherein the return value specifies whether the multimodal interface workflow has concluded.

14. A computer-implemented method for client-side implementation of an interface automation language at runtime, the computer-implemented method comprising:

constructing, on the client-side, an agent specification, making the agent specification available for server-side translation into an intermediate representation, wherein the agent specification is configured to automate a multimodal interface workflow;

receiving, on the client-side, the intermediate representation;

detecting, on the client-side, one or more agent functions in the intermediate representation;

generating, on the client-side, one or more agent calls based on the agent functions;

issuing, on the client-side, the agent calls to an agent on the server-side, and, in response, receiving, on the client-side, at least one runtime actuation function from the agent; and

translating, on the client-side, the runtime actuation function into at least one runtime actuation command, wherein the runtime actuation command triggers at least one machine-actuated action as a runtime synthetic action that automates the multimodal interface workflow.

15. The computer-implemented method of claim 14 , wherein the agent functions include built-in functions, planner functions, and workflow functions.

16. The computer-implemented method of claim 15 , wherein the built-in functions include answerQuestionAboutScreen, goToURL, typeIntoElement, click, type, wait, goToSong, compose, answerTrueFalseQuestionAboutScreen, composeAndType, getCurrentDate, isVisible, keydown, print, scroll, and spotlight.

17. The computer-implemented method of claim 15 , wherein the planner functions include act, fillform, and pickdate.

18. The computer-implemented method of claim 17 , further comprising invoking an observation logic in response to detecting the act planner function.

19. The computer-implemented method of claim 18 , sending, with the observation logic, one or more interface screenshots, an action history, and a task description to the agent.

20. A non-transitory computer readable storage medium impressed with computer program instructions for client-side implementation of an interface automation language at runtime, the instructions, when executed on a processor, implement a method comprising:

constructing, on the client-side, an agent specification, making the agent specification available for server-side translation into an intermediate representation, wherein the agent specification is configured to automate a multimodal interface workflow;

receiving, on the client-side, the intermediate representation;

detecting, on the client-side, one or more agent functions in the intermediate representation;

generating, on the client-side, one or more agent calls based on the agent functions;

issuing, on the client-side, the agent calls to an agent on the server-side, and, in response, receiving, on the client-side, at least one runtime actuation function from the agent; and

translating, on the client-side, the runtime actuation function into at least one runtime actuation command, wherein the runtime actuation command triggers at least one machine-actuated action as a runtime synthetic action that automates the multimodal interface workflow.

Assignments (3)

Continuity (8)

References Cited (125)

Cited By (1)

US 12,675,385