User interface automation using natural language

1 . A computer-implemented method, comprising:

receiving, by a computing device, during a teaching phase of an automation, a demonstration of the automation to be performed on an application;

detecting, on the application, during the teaching phase of the automation, objects and labels associated with the demonstration, wherein each object of the objects is associated with a corresponding label of the labels;

transforming, by the computing device, during the teaching phase of the automation, the demonstration into natural language instructions based on the detecting of the objects and the labels;

storing the natural language instructions in a knowledge representation dataset; and

semantically selecting, during a runtime phase of the automation, an object of the objects based on a large language model (LLM), wherein the semantically selecting of the object comprises:

detecting a change in the corresponding label of the object, wherein the changed corresponding label is associated with a natural language instruction of the stored natural language instructions;

obtaining a plurality of possible objects among the objects, wherein the obtained plurality of possible objects corresponds to the changed corresponding label;

providing, to the LLM, the natural language instruction and the obtained plurality of possible objects;

receiving, from the LLM, an indication of the object that most likely corresponds to the natural language instruction among the obtained plurality of possible objects; and

selecting the object based on the receiving of the indication of the object, wherein

the semantically selecting is reflective of an intention captured in the natural language instruction of the stored natural language instructions, despite the change in the corresponding label of the object.

2 . The computer-implemented method of claim 1 , wherein the automation is performed by PbD (programming by demonstration).

3 . The computer-implemented method of claim 1 , further comprising:

detecting the objects and the labels by performing a page understanding of a page of the application to generate a dataset representative of an original state of the objects available on the page of the application.

4 . The computer-implemented method of claim 1 , further comprising:

performing a page understanding on a page of the application to generate a dataset representative of a current state of the objects available on the page of the application; and

detecting the change in the corresponding label of the object based on the performing of the page understanding.

5 . The computer-implemented method of claim 1 , wherein the application is a plug-in module configured to be plugged into a web, a desktop, or a mobile application.

6 . A system, comprising:

a processor configured to:

receive, during a teaching phase of an automation, a demonstration of the automation to be performed on an application;

detect, on the application, during the teaching phase of the automation, objects and labels associated with the demonstration, wherein each object of the objects is associated with a corresponding label of the labels;

transform, during the teaching phase of the automation, the demonstration into natural language instructions based on the detection of the objects and the labels;

store the natural language instructions in a knowledge representation dataset; and

semantically select, during a runtime phase of the automation, an object of the objects based on a large language model (LLM), wherein the semantic selection of the object comprises:

detection of a change in the corresponding label of the object, wherein the changed corresponding label is associated with a natural language instruction of the stored natural language instructions;

obtainment of a plurality of possible objects among the objects, wherein the obtained plurality of possible objects corresponds to the changed corresponding label;

provision, to the LLM, of the natural language instruction and the obtained plurality of possible objects;

reception, from the LLM, of an indication of the object that most likely corresponds to the natural language instruction among the obtained plurality of possible objects; and

selection of the object based on the reception of the indication of the object, wherein

the semantic selection is reflective of an intent of the natural language instruction of the stored natural language instructions, despite the change in the corresponding label of the object.

7 . The system of claim 6 , wherein the detected objects are user interface elements.

8 . The system of claim 6 , wherein the automation is performed by PbD (programming by demonstration).

9 . The system of claim 6 , wherein the application is a plug-in module configured to be plugged into a web, a desktop, or a mobile application.

10 . A non-transitory computer-readable storage medium storing computer-executable instructions, which when executed by a computer system causes the computer system to:

receive, during a teaching phase of an automation, a demonstration of the automation to be performed on an application;

detect, on the application, during the teaching phase of the automation, objects and labels associated with the demonstration, wherein each object of the objects is associated with a corresponding label of the labels;

transform, during the teaching phase of the automation, the demonstration into natural language instructions based on the detection of the objects and the labels;

store the natural language instructions in a knowledge representation dataset; and

semantically select, during a runtime phase of the automation, an object of the objects based on a large language model (LLM), wherein the semantic selection of the object comprises:

detection of a change in the corresponding label of the object, wherein the changed corresponding label is associated with a natural language instruction of the stored natural language instructions;

obtainment of a plurality of possible objects among the objects, wherein the obtained plurality of possible objects corresponds to the changed corresponding label;

provision, to the LLM, of the natural language instruction and the obtained plurality of possible objects;

reception, from the LLM, of an indication of the object that most likely corresponds to the natural language instruction among the obtained plurality of possible objects; and

selection of the object based on the reception of the indication of the object wherein

the semantic selection is reflective of an intention captured in the natural language instruction of the stored natural language instructions, despite the change in the corresponding label of the object.

11 . The non-transitory computer-readable storage medium of claim 10 , wherein the automation is performed by PbD (programming by demonstration).

12 . The non-transitory computer-readable storage medium of claim 10 , wherein the application is a plug-in module configured to be plugged into a web, a desktop, or a mobile application.

13 . A computer-implemented method, comprising:

receiving, by a computing device, during a teaching phase of an automation, a demonstration of the automation to be performed on an application;

transforming, by the computing device, during the teaching phase of the automation, the demonstration into natural language instructions;

storing the natural language instructions in a knowledge representation dataset;

semantically selecting, during a runtime phase of the automation, an object of objects associated with the demonstration based on a large language model (LLM), wherein the semantically selecting of the object comprises:

detecting a change in a label of the object, wherein the changed label is associated with a natural language instruction of the stored natural language instructions;

obtaining a plurality of possible objects among the objects, wherein the obtained plurality of possible objects corresponds to the changed label;

providing, to the LLM, the natural language instruction and the obtained plurality of possible objects;

receiving, from the LLM, an indication of the object that most likely corresponds to the natural language instruction among the obtained plurality of possible objects; and

selecting the object based on the receiving of the indication of the object, wherein

the semantically selecting is reflective of an intention captured in the natural language instruction of the stored natural language instructions, despite the change in the label of the object; and

modifying the automation based on a new natural language user input.

14 . The computer-implemented method of claim 13 , wherein the modifying of the automation is performed to update, replace, or add one or more natural language instructions to the automation.

15 . A system, comprising:

a knowledge representation dataset comprising natural language instructions of an automation; and

a processor configured to:

receive the knowledge representation dataset; and

semantically select, during a runtime phase the automation, an object of objects based on a large language model (LLM), wherein the semantic selection of the object comprises:

detection of a change in a label of the object, wherein the changed label is associated with a natural language instruction of the natural language instructions;

obtainment of a plurality of possible objects among the objects, wherein the obtained plurality of possible objects corresponds to the changed label;

provision, to the LLM, of the natural language instruction and the obtained plurality of possible objects;

reception, from the LLM, of an indication of the object that most likely corresponds to the natural language instruction among the obtained plurality of possible objects; and

selection of the object based on the reception of the indication of the object, wherein

the semantic selection is reflective of an intent of the natural language instruction of the natural language instructions, despite the change in the label of the object.

16 . The system of claim 15 , wherein the processor is further configured to:

receive, during a teaching phase of the automation, a demonstration of the automation to be performed on an application;

detect, on the application, during the teaching phase of the automation, the objects and labels associated with the demonstration, wherein each object of the objects is associated with a corresponding label of the labels; and

transform, during the teaching phase of the automation, the demonstration into the natural language instructions representative of the automation to be performed on the application.