Software robots for programmatically controlling computer programs to perform tasks
A system comprising at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising: obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.
1. A system for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the system comprising:
at least one hardware processor;
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program at least in part by:
obtaining a first representation of active GUI elements of the first application program from an operating system,
generating, based on the first representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program,
obtaining a second representation of active GUI elements of the second application program from a source other than the operating system, and
generating, based on the second representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second active GUI element of the second application program;
controlling the first application program to perform the first sub-task using the object hierarchy at least in part by:
using the first object corresponding to the first active GUI element of the first application program to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and
controlling the second application program to perform the second sub-task using the object hierarchy at least in part by:
using the second object corresponding to the second active GUI element of the second application program to cause, via the second active GUI element, the second application program to at least partially perform a second action in furtherance of the second sub-task.
2. The system of claim 1 , wherein the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one hardware processor to:
obtain the second representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.
3. The system of claim 1 , wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.
4. The system of claim 1 , wherein:
controlling the first application program to perform the first sub-task further comprises accessing, in the first portion of the object hierarchy, the first object corresponding to the first active GUI element of the first application program, and
controlling the second application program to perform the second sub-task further comprises accessing, in the second portion of the object hierarchy, the second object corresponding to the second active GUI element of the second application program.
5. The system of claim 4 , wherein accessing the first object comprises refreshing the first portion of the object hierarchy and accessing the second object comprises refreshing the second portion of the object hierarchy.
6. The system of claim 4 , wherein accessing the first object comprises refreshing the object hierarchy.
7. The system of claim 4 , wherein accessing the second object comprises refreshing the object hierarchy.
8. A method for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the method comprising:
using at least one computer hardware processor to perform:
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs at least in part by:
obtaining a first representation of active GUI elements of the first application program from an operating system,
generating, based on the first representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program,
obtaining a second representation of active GUI elements of the second application program from a source other than the operating system, and
generating, based on the second representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second active GUI element of the second application program;
controlling the first application program to perform the first sub-task using the generated object hierarchy at least in part by:
using the first object corresponding to the first active GUI element of the first application program to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and
controlling the second application program to perform the second sub-task using the object hierarchy at least in part by:
using the second object corresponding to the second active GUI element of the second application program to cause, via the second active GUI element, the second application program to at least partially perform a second action in furtherance of the second sub-task.
9. The method of claim 8 , wherein the second application program is an Internet browser program, and obtaining a second representation comprises:
obtaining the second representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.
10. The method of claim 8 , wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.
11. The method of claim 8 , wherein:
controlling the first application program to perform the first sub-task further comprises accessing, in the first portion of the object hierarchy, the first object corresponding to the first active GUI element of the first application program, and
controlling the second application program to perform the second sub-task further comprises accessing, in the second portion of the object hierarchy, the second object corresponding to the second active GUI element of the second application program.
12. The method of claim 11 , wherein accessing the first object comprises refreshing the first portion of the object hierarchy and accessing the second object comprises refreshing the second portion of the object hierarchy.
13. The method of claim 11 , wherein accessing the first object comprises refreshing the object hierarchy.
14. The method of claim 11 , wherein accessing the second object comprises refreshing the object hierarchy.
15. A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to perform a method for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the method comprising:
generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs at least in part by:
obtaining a first representation of active GUI elements of the first application program from an operating system,
generating, based on the first representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program, the first portion of the object hierarchy comprising a first object corresponding to a first active GUI element of the first application program,
obtaining a second representation of active GUI elements of the second application program from a source other than the operating system, and
generating, based on the second representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, the second portion of the object hierarchy comprising a second object corresponding to a second active GUI element of the second application program;
controlling the first application program to perform the first sub-task using the generated object hierarchy at least in part by:
using the first object corresponding to the first active GUI element of the first application program to cause, via the first active GUI element, the first application program to at least partially perform a first action in furtherance of the first sub-task; and
controlling the second application program to perform the second sub-task using the object hierarchy at least in part by:
using the second object corresponding to the second active GUI element of the second application program to cause, via the second active GUI element, the second application program to at least partially perform a second action in furtherance of the second sub-task.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the second application program is an Internet browser program, and obtaining a second representation comprises:
obtaining the second representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.
18. The non-transitory computer-readable storage medium of claim 15 , wherein:
controlling the first application program to perform the first sub-task further comprises accessing, in the first portion of the object hierarchy, the first object corresponding to the first active GUI element of the first application program, and
controlling the second application program to perform the second sub-task further comprises accessing, in the second portion of the object hierarchy, the second object corresponding to the second active GUI element of the second application program.
19. The non-transitory computer-readable storage medium of claim 18 , wherein accessing the first object comprises refreshing the first portion of the object hierarchy and accessing the second object comprises refreshing the second portion of the object hierarchy.
20. The non-transitory computer-readable storage medium of claim 18 , wherein accessing the first object comprises refreshing the object hierarchy.