Granted Patent B2

US 12,566,913 · App. 18/909,068 · Granted Mar 3, 2026

Inventors: Rohan Bavishi (San Francisco, CA); Lina Lukyantseva (San Francisco, CA); Shaya Zarkesh (San Francisco, CA); David Luan (San Francisco, CA); Basil Safwat (San Francisco, CA); Amelia Wattenberger (San Francisco, CA); Kadhir Manickam (San Francisco, CA); Inigo Beitia Arevalo (San Francisco, CA); James Lu (San Francisco, CA); Omkar Savant (San Francisco, CA); Zach Brock (San Francisco, CA); Jacob van Gogh (San Francisco, CA); Rick Liu (San Francisco, CA); Deepak Moparthi (San Francisco, CA); Claire Pajot (San Francisco, CA); Joe Gershenson (San Francisco, CA); Arushi Somani (San Francisco, CA); Armaan Goel (San Francisco, CA); Kevin Keller (San Francisco, CA); Erich Elsen (San Francisco, CA); Curtis Hawthorne (San Francisco, CA)

Assignee: Anthropic, PBC

G06F40/166G06F3/0481G06F3/0484G06F16/951G06F40/174G06F40/284G06N3/0455G06N3/091G06N5/04G06N20/00G06V10/7715G06V10/774G06V10/803G06V10/82G06V20/40G06V30/19147G06V30/41G06F9/451

Abstract

Claims (23)

1 . A system for interface automation, comprising:

at least one central processing unit:

a memory device storing programming and data constructs that when executed by the at least one processing unit, cause the system to configure an agent;

the agent configured to:

process an input including multimodal data that specifies an interface workflow including contextual metadata and a state of the interface prior to execution of the interface workflow, wherein the multimodal data includes at least a combination of a natural language description and a prescriptive command; and

segment the interface workflow into a plurality of sub-tasks wherein each sub-task depends on the completion of a preceding sub-task;

generate an output, responsive to the multimodal input data, that specifies a sequence of actuation commands expressed in a custom domain-specific language (DSL) that supports both model calls and action executions, wherein the sequence of actuation commands triggers one or more machine-actuated actions that replicate user-actuated actions on the interface and cause automation of the interface workflow by translating model instructions into real web or application events, including at least localization;

actuate the sequence of multimodal actuation commands by an actuator, wherein the actuator is configured to receive the sequence of actuation commands from the agent, and to perform the machine-actuated actions based on the sequence of actuation commands as synthetic actions that automate the interface workflow; and

provide feedback including interface screenshots and action histories for iterative workflow execution and refinement.

2 . The system of claim 1 , wherein the natural language description is a natural language description of the interface workflow provided by the user and processed by a trained multimodal transformer model to identify intent and specific interface elements involved.

3 . The system of claim 1 , wherein the prescriptive command translates the interface workflow into one or more functions and one or more parameters using a custom-designed Domain Specific Language (DSL).

4 . The system of claim 3 , wherein the parameters are key-value pairs.

5 . The system of claim 3 , wherein the parameters include descriptions of the functions.

6 . The system of claim 1 , wherein the input includes the state of the interface prior to the execution of the interface workflow captured as structured contextual metadata, wherein said metadata is dynamically updated based on real-time interface interactions.

7 . The system of claim 1 , wherein the state of the interface prior to the execution of the interface workflow includes one or more snapshots of the interface processed as image tokens using a vision transformer module.

8 . The system of claim 1 , wherein the state of the interface prior to the execution of the interface workflow includes metadata about the interface.

9 . The system of claim 1 , wherein the state of the interface prior to the execution of the interface workflow receives one or more hints from the user as the input that contextualize the interface workflow.

10 . The system of claim 1 , wherein the state of the interface prior to the execution of the interface workflow includes a description of the interface workflow.

11 . The system of claim 1 , wherein the interface workflow includes a plurality of sub-tasks.

12 . The system of claim 11 , wherein a current sub-task in the plurality of sub-tasks is a result of executing one or more preceding sub-tasks in the plurality of sub-tasks.

13 . The system of claim 12 , wherein the state of the interface prior to the execution of the current sub-task includes one or more snapshots of the interface corresponding to the current sub-task, one or more snapshots of the interface corresponding to the preceding sub-tasks, and one or more sequences of actuation commands corresponding to the preceding sub-tasks.

14 . The system of claim 1 , wherein the actuator is implemented by an actuation layer configured to translate the sequence of actuation commands into real-time synthetic interactions with interface elements, wherein the actuator is configured to receive the sequence of actuation commands from the agent, and to perform the machine-actuated actions based on the sequence of actuation commands as synthetic actions that automate the interface workflow.

15 . The system of claim 1 , wherein the user-actuated actions include clicks, hovers, scrolls, picks, text entries, and form fills.

Assignments (3)

Continuity (9)

References Cited (142)