Generate a script to automate a task associated with a webpage
A query that includes one or more variables is received. The one or more variables correspond to one or more interactive elements. A large language model is utilized to generate a query response that associates one or more variables included in the query to the one or more interactive elements. A script is generated utilizing the query response that associates the one or more variables included in the query to the one or more interactive elements.
1 . A method, comprising:
receiving a query that includes one or more variables, wherein the one or more variables correspond to a first set of one or more interactive elements;
processing the first set of one or more interactive elements to determine one or more corresponding parameters associated with the one or more interactive elements;
providing to a cloud service the query that includes the one or more variables and the one or more corresponding parameters associated with the one or more interactive elements included in the first set, wherein the cloud service generates a prompt based on the query and the one or more corresponding parameters associated with the one or more interactive elements and provides the prompt to a large language model to generate a query response that maps the one or more variables included in the query to the first set of the one or more interactive elements; and
generating a script utilizing the one or more variables included in the query based on the query response received from the large language model that maps the one or more variables included in the query to the first set of the one or more interactive elements.
2 . The method of claim 1 , wherein the first set of the one or more interactive elements are associated with a webpage.
3 . The method of claim 1 , wherein the first set of the one or more interactive elements are associated with an application.
4 . The method of claim 3 , further comprising obtaining user interface content associated with the application.
5 . The method of claim 4 , wherein the user interface content associated with the application is extracted from a user interface tree associated with the application.
6 . The method of claim 5 , wherein the user interface content associated with the application is extracted by a user interface content retrieval service associated with a device on which the application is installed.
7 . The method of claim 6 , wherein the extracted user interface content associated with the application includes a user interface layout, screen content, or a screenshot.
8 . The method of claim 6 , wherein the user interface content retrieval service is installed on the device.
9 . The method of claim 6 , wherein the user interface content retrieval service is installed on a second device separate from the device.
10 . The method of claim 4 , further comprising processing the user interface content associated with the application into a consumable format.
11 . The method of claim 10 , wherein the consumable format is XML, JSON or a screenshot.
12 . The method of claim 10 , wherein the processed user interface content associated with the application includes location information for the first set of the one or more interactive elements.
13 . The method of claim 10 , wherein the one or more corresponding parameters associated with the one or more interactive elements include a role, a name, and/or a tag.
14 . The method of claim 1 , wherein the large language model is configured to generate the query response in response to receiving the prompt.
15 . The method of claim 1 , further comprising receiving the query response from the large language model via the cloud service.
16 . A system, comprising:
a processor configured to:
receive a query that includes one or more variables, wherein the one or more variables correspond to a first set of one or more interactive elements;
process the first set of one or more interactive elements to determine one or more corresponding parameters associated with the one or more interactive elements;
provide to a cloud service the query that includes the one or more variables and the one or more corresponding parameters associated with the one or more interactive elements included in the first set, wherein the cloud service is configured to generate a prompt based on the query and the one or more corresponding parameters associated with the one or more interactive elements and provide the prompt to a large language model to generate a query response that maps one or more variables included in the query to the first set of the one or more interactive elements; and
generate a script utilizing the one or more variables included in the query based on the query response received from the large language model that maps the one or more variables included in the query to the first set of one or more interactive elements; and
a memory coupled to the processor and configured to provide the processor with instructions.
17 . The system of claim 16 , wherein the first set of the one or more interactive elements are associated with an application.
18 . The system of claim 17 , wherein the processor is configured to obtain user interface content associated with the application that is extracted from a user interface tree associated with the application.
19 . The system of claim 18 , wherein the processor is further configured to process the user interface content associated with the application into a consumable format.
20 . The system of claim 19 , wherein the consumable format is XML, JSON or a screenshot.
21 . The system of claim 19 , wherein the processed user interface content associated with the application includes location information for the first set of the one or more interactive elements.
22 . The system of claim 19 , wherein the one or more corresponding parameters associated with the one or more interactive elements include a role, a name, and/or a tag.
23 . The system of claim 17 , wherein the processor is further configured to obtain user interface content associated with the application.
24 . The system of claim 23 , wherein the user interface content associated with the application is extracted from a user interface tree associated with the application.
25 . The system of claim 24 , wherein the user interface content associated with the application is extracted by a user interface content retrieval service associated with a device on which the application is installed.
26 . The system of claim 25 , wherein the extracted user interface content associated with the application includes a user interface layout, screen content, or a screenshot.
27 . The system of claim 25 , wherein the user interface content retrieval service is installed on the device.
28 . The system of claim 25 , wherein the user interface content retrieval service is installed on a second device separate from the device.
29 . The system of claim 16 , wherein the first set of the one or more interactive elements are associated with a webpage.
30 . The system of claim 16 , wherein the large language model is configured to generate the query response in response to receiving the prompt.
31 . The system of claim 16 , wherein the processor is further configured to receive the query response from the large language model via the cloud service.
32 . A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving a query that includes one or more variables, wherein the one or more variables correspond to a first set of one or more interactive elements;
processing the first set of one or more interactive elements to determine one or more corresponding parameters associated with the one or more interactive elements;
providing to a cloud service the query that includes the one or more variables and the one or more corresponding parameters associated with the one or more interactive elements included in the first set, wherein the cloud service generates a prompt based on the query and the one or more corresponding parameters associated with the one or more interactive elements and provides the prompt to a large language model to generate a query response that maps the one or more variables included in the query to the first set of the one or more interactive elements; and
generating a script utilizing the one or more variables included in the query based on the query response received from the large language model that maps the one or more variables included in the query to the first set of one or more interactive elements.