Instruction interpretation for web task automation
View Patent ↗A method of generating an instruction performance skeleton employs an instruction unit configured to receive a natural language instruction. From the natural language instruction, a sequence of clauses may be extracted. The instruction unit then determines a target website or websites on which to perform the task. The object models of the target website are generated. A comparison of the sequence of actions to the object model and its labelling hierarchical class structure is performed. Based on this comparison, an instruction performance skeleton is generated. In future, on the basis of a further natural language instruction that is similar to the previous natural language instruction, the instruction performance skeleton may be modified to generate a playback performance skeleton to arrange performance of a task.
1. A method comprising:
receiving a natural language instruction describing a sequence of actions representative of a manner in which a human would interact with a web browser to perform the actions on web elements of at least one web page, wherein an instruction performance skeleton is based on the natural language instruction;
storing the instruction performance skeleton in a database, wherein the instruction performance skeleton is representative of performance of the actions and includes respective key-value pairs associated with each of the actions; and
concurrently performing a first action of the actions in a first webview of the web browser in parallel with a second action of the actions in a different second webview of the web browser, wherein:
the first webview comprises a first object model of the at least one web page and the second webview comprises a second object model of the at least one web page, and
the first object model comprises a first plurality of the web elements and the second object model comprises a second plurality of the web elements.
2. The method of claim 1 , wherein at least one of the first object model and the second object model is a Document Object Model (DOM).
3. The method of claim 2 , wherein the DOM has a hierarchical tree structure having branches.
4. The method of claim 3 , wherein the DOM has a plurality of labels.
5. The method of claim 4 , wherein at least one label is an Accessible Rich Internet Application label.
6. The method of claim 1 , wherein the instruction performance skeleton further includes a conditionality, wherein the conditionality is based on the natural language instruction, and wherein the conditionality is based on the performance of the actions.
7. The method of claim 1 , wherein the at least one web page comprises at least two web pages.
8. The method of claim 1 , wherein at least one web element comprises at least one of a class and a tag.
9. The method of claim 1 , wherein at least one webview comprises a mutation observer.
10. The method of claim 1 , wherein at least one of the first action and the second action is one of a right click, a left click, and a typing action.
11. A method comprising:
receiving a natural language instruction describing a sequence of actions representative of a manner in which a human would interact with a web browser to perform the actions on web elements of at least one web page, wherein an instruction performance skeleton is based on the natural language instruction;
storing the instruction performance skeleton in a database, wherein the instruction performance skeleton is representative of performance of the actions and includes respective key-value pairs associated with each of the actions;
performing a first action of the actions in a first webview of the web browser, the first webview comprising a first object model of the at least one web page, the first object model comprising a first plurality of the web elements;
receiving an update message related to the performance of the first action, the update message including information about at least one change to the first object model of the at least one web page caused by the performance of the first action; and
based on the at least one change, performing a second action of the actions in a different second webview of the web browser, the second webview comprising a second object model of the at least one web page, the second object model comprising a second plurality of the web elements, and the performance of the second action causing at least one change to the second object model of the at least one web page.
12. The method of claim 11 , wherein at least one of the first the object model and the second object model is a Document Object Model (DOM).
13. The method of claim 12 , wherein the DOM has a hierarchical tree structure having branches.
14. The method of claim 11 , wherein at least one web element comprises at least one of a class and a tag.
15. The method of claim 11 , wherein the receiving the update message comprises receiving the update message from a headless browser.
16. The method of claim 11 , wherein the receiving the update message comprises receiving the update message from an electronic device hosting the web browser implementing the webview.
17. The method of claim 11 , wherein the first action is one of a right click, a left click, and a typing action.
18. The method of claim 11 , wherein the second action is one of a right click, a left click, and a typing action.
19. The method of claim 13 , wherein the DOM has a plurality of labels.
20. The method of claim 11 , wherein at least one webview comprises a mutation observer.