Granted Patent B2

US 12,651,033 · App. 18/889,959 · Granted Jun 9, 2026

Inventors: Shuhao Zhang (Cupertino, CA); Qi Zhai (Hillsborough, CA); Daniel Lawrence Schafer (Portola Valley, CA); Pavlo Dudka (Mountain View, CA); Mingyang Li (Bellevue, WA); Urvish Dipakkumar Thakker (Sunnyvale, CA)

Assignee: Tiny Fish Inc.

G06F16/986G06F16/9532G06F16/9538

Abstract

Claims (58)

1 . A method, comprising:

receiving, by a web browser application associated with a client device, a first instance of a query and processed webpage content, wherein the processed webpage content is processed by determining corresponding information associated with a plurality of user interface web elements associated with a webpage, wherein the corresponding information associated with the plurality of user interface web elements associated with the webpage includes one or more of a corresponding role, a corresponding name, and/or a corresponding tag for a plurality of web elements associated with the webpage;

generating a prompt for a large language model based on the received query and the received processed webpage content;

receiving a query response from the large language model that is generated in response to providing the prompt to the large language model, wherein the query response preserves a mapping of response nodes to corresponding user interface web elements;

generating inference patterns based on the query response;

storing the inference patterns in an inference patterns store;

providing the query response to a client software development kit (SDK) included in the browser application, wherein the client SDK includes one or more application programming interface(s) (APIs) that enable object-oriented programming interfaces to be generated based on the query response and provide functionality to interact with the web elements, wherein the APIs are supported by one or more programming languages;

receiving, by the web browser application associated with the client device, a second instance of the query and the processed webpage content;

determining, based on the mapping of response nodes to corresponding user interface web elements, that the query and the processed webpage content have been previously received;

in response to a determination that query and the processed webpage content have been previously received:

retrieving the inference patterns that are based on the query response received from the large language model from the inference pattern store; and

generating a second query response utilizing the inference patterns from the inference pattern store; and

generating, using the one or more APIs included in the client SDK, an automated task for the webpage utilizing the second query response.

2 . The method of claim 1 , wherein the query includes one or more variables to be used for a script associated with the webpage.

3 . The method of claim 2 , wherein the one or more variables correspond to one or more specific web elements associated with the webpage.

4 . The method of claim 1 , wherein the corresponding role describes a role of a particular web element of the plurality of web elements in an accessibility tree.

5 . The method of claim 1 , wherein the corresponding name represents the name of a particular web element of the plurality of web elements as specified in an original webpage accessibility tree.

6 . The method of claim 1 , wherein the corresponding tag denotes an original html tag of a particular web element of the plurality of web elements.

7 . The method of claim 1 , further comprising providing the prompt, the received query, and the received processed webpage content to the large language model.

8 . A system, comprising:

a processor configured to:

receive, by a web browser application associated with a client device, a first instance of a query and processed webpage content, wherein the processed webpage content is processed by determining corresponding information associated with a plurality of user interface web elements associated with a webpage, wherein the corresponding information associated with the plurality of user interface web elements associated with the webpage includes one or more of a corresponding role, a corresponding name, and/or a corresponding tag for a plurality of web elements associated with the webpage;

generate a prompt for a large language model based on the received query and the received processed webpage content;

receive a query response from the large language model that is generated in response to providing the prompt to the large language model, wherein the query response preserves a mapping of response nodes to corresponding user interface web elements;

generate inference patterns based on the query response;

store the inference patterns in an inference patterns store;

provide the query response to a client software development kit (SDK) included in the browser application, wherein the client SDK includes one or more application programming interface(s) (APIs) that enable object-oriented programming interfaces to be generated based on the query response and provide functionality to interact with the web elements, wherein the APIs are supported by one or more programming languages;

receive, by the web browser application associated with the client device, a second instance of the query and the processed webpage content;

determine, based on the mapping of response nodes to corresponding user interface web elements, that the query and the processed webpage content have been previously received;

in response to a determination that query and the processed webpage content have been previously received:

retrieve the inference patterns that are based on the query response received from the large language model from the inference pattern store; and

generate a second query response utilizing the inference patterns from the inference pattern store; and

generating, using the one or more APIs included in the client SDK, an automated task for the webpage utilizing the second query response; and

a memory coupled to the processor and configured to provide the processor with instructions.

9 . The system of claim 8 , wherein the query includes one or more variables to be used for a script associated with the webpage, wherein the one or more variables correspond to one or more specific web elements associated with the webpage.

10 . The system of claim 8 , wherein the corresponding role describes a role of a particular web element of the plurality of web elements in an accessibility tree.

11 . The system of claim 8 , wherein the corresponding name represents the name of a particular web element of the plurality of web elements as specified in an original webpage accessibility tree.

12 . The system of claim 8 , wherein the corresponding tag denotes an original html tag of a particular web element of the plurality of web elements.

13 . The system of claim 8 , wherein the processor is configured to provide the prompt, the received query, and the received processed webpage content to the large language model.

14 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

receiving, by a web browser application associated with a client device, a first instance of a query and processed webpage content, wherein the processed webpage content is processed by determining corresponding information associated with a plurality of user interface web elements associated with a webpage, wherein the corresponding information associated with the plurality of user interface web elements associated with the webpage includes one or more of a corresponding role, a corresponding name, and/or a corresponding tag for a plurality of web elements associated with the webpage;

generating a prompt for a large language model based on the received query and the received processed webpage content;

receiving a query response from the large language model that is generated in response to providing the prompt to the large language model, wherein the query response preserves a mapping of response nodes to corresponding user interface web elements;

generating inference patterns based on the query response;

storing the inference patterns in an inference patterns store;

providing the query response to a client software development kit (SDK) included in the browser application, wherein the client SDK includes one or more application programming interface(s) (APIs) that enable object-oriented programming interfaces to be generated based on the query response and provide functionality to interact with the web elements, wherein the APIs are supported by one or more programming languages;

receiving, by the web browser application associated with the client device, a second instance of the query and the processed webpage content;

determining, based on the mapping of response nodes to corresponding user interface web elements, that the query and the processed webpage content have been previously received;

in response to a determination that query and the processed webpage content have been previously received:

retrieving the inference patterns that are based on the query response received from the large language model from the inference pattern store; and

generating a second query response utilizing the inference patterns from the inference pattern store; and

generating, using the one or more APIs included in the client SDK, an automated task for the webpage utilizing the second query response.

15 . The computer program product of claim 14 , wherein the query includes one or more variables to be used for a script associated with the webpage.

16 . The computer program product of claim 15 , wherein the one or more variables correspond to one or more specific web elements associated with the webpage.

17 . The computer program product of claim 14 , wherein the corresponding role describes a role of a particular web element of the plurality of web elements in an accessibility tree.

18 . The computer program product of claim 14 , wherein the corresponding name represents the name of a particular web element of the plurality of web elements as specified in an original webpage accessibility tree.

19 . The computer program product of claim 14 , wherein the corresponding tag denotes an original html tag of a particular web element of the plurality of web elements.

20 . The computer program product of claim 14 , further comprising instructions for providing the prompt, the received query, and the received processed webpage content to the large language model.

Continuity (3)

References Cited (100)