Generate a script to automate a task associated with a webpage
A query that includes one or more variables is received. The one or more variables correspond to one or more interactive elements. A large language model is utilized to generate a query response that associates one or more variables included in the query to the one or more interactive elements. A script is generated utilizing the query response that associates the one or more variables included in the query to the one or more interactive elements.
1. A method, comprising:
receiving a query that includes one or more variables, wherein the one or more variables correspond to a first set of one or more interactive elements;
utilizing a large language model to generate a query response that maps the one or more variables included in the query to the first set of one or more interactive elements;
generating a script utilizing the one or more variables included in the query based on the query response received from the large language model that maps the one or more variables included in the query to the first set of one or more interactive elements; and
updating the generated script utilizing an updated query response in response to determining that the generated script is nonfunctional, wherein the updated query response maps the one or more variables included in the query to a second set of one or more interactive elements.
2. The method of claim 1 , wherein the first set of one or more interactive elements are associated with a webpage.
3. The method of claim 1 , wherein the first set of one or more interactive elements are associated with an application.
4. The method of claim 3 , further comprising obtaining user interface content associated with the application.
5. The method of claim 4 , wherein the user interface content associated with the application is extracted from a user interface tree associated with the application.
6. The method of claim 5 , wherein the user interface content associated with the application is extracted by a user interface content retrieval service associated with a device on which the application is installed.
7. The method of claim 6 , wherein the extracted user interface content associated with the application includes a user interface layout, screen content, or a screenshot.
8. The method of claim 6 , wherein the user interface content retrieval service is installed on the device.
9. The method of claim 6 , wherein the user interface content retrieval service is installed on a second device separate from the device.
10. The method of claim 4 , further comprising processing the user interface content associated with the application into a consumable format.
11. The method of claim 10 , wherein the consumable format is XML, JSON or a screenshot.
12. The method of claim 10 , wherein the processed user interface content associated with the application includes location information for the first set of one or more interactive elements.
13. The method of claim 10 , wherein the processed user interface content associated with the application includes a role, a name, and/or a tag for the first set of one or more interactive elements.
14. The method of claim 1 , wherein utilizing the large language model to generate the query response that maps one or more variables included in the query to the first set of one or more interactive elements includes providing to the large language model the query and processed user interface content associated with an application.
15. The method of claim 14 , wherein utilizing the large language model to generate the query response that maps one or more variables included in the query to the first set of one or more interactive elements further includes receiving the query response from the large language model.
16. A system, comprising:
a processor configured to:
receive a query that includes one or more variables, wherein the one or more variables correspond to a first set of one or more interactive elements;
utilize a large language model to generate a query response that maps one or more variables included in the query to the first set of one or more interactive elements;
generate a script utilizing the one or more variables included in the query based on the query response received from the large language model that maps the one or more variables included in the query to the first set of one or more interactive elements; and
update the generated script utilizing an updated query response in response to determining that the generated script is nonfunctional, wherein the updated query response maps the one or more variables included in the query to a second set of one or more interactive elements; and
a memory coupled to the processor and configured to provide the processor with instructions.
17. The system of claim 16 , wherein the first set of one or more interactive elements are associated with an application.
18. The system of claim 17 , wherein the processor is configured to obtain user interface content associated with the application that is extracted from a user interface tree associated with the application.
19. The system of claim 18 , wherein the processor is further configured to process the user interface content associated with the application into a consumable format.
20. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving a query that includes one or more variables, wherein the one or more variables correspond to a first set of one or more interactive elements;
utilizing a large language model to generate a query response that maps the one or more variables included in the query to the first set of one or more interactive elements;
generating a script utilizing the one or more variables included in the query based on the query response received from the large language model that maps the one or more variables included in the query to the first set of one or more interactive elements; and
updating the generated script utilizing an updated query response in response to determining that the generated script is nonfunctional, wherein the updated query response maps the one or more variables included in the query to a second set of one or more interactive elements.