Software robots for programmatically controlling computer programs to perform tasks

1. A system for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the system comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising:

identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;

generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program,

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by:

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element; and

after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; and

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by:

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element; and

after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.

2. The system of claim 1 , wherein refreshing the object hierarchy comprises:

refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one object from the object hierarchy.

3. The system of claim 1 , wherein refreshing the object hierarchy comprises:

refreshing only a subset of objects in the object hierarchy.

4. The system of claim 1 , wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program.

5. The system of claim 1 , wherein refreshing the object hierarchy comprises:

refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object.

6. The system of claim 1 , wherein refreshing the object hierarchy comprises:

adding a new object to the object hierarchy, wherein the new object is associated with an active GUI element of the first application program.

7. The system of claim 1 , wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

8. The system of claim 1 , further comprising a display, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

displaying a GUI of the first application program on the display during the performance of at least some of the sequence of actions.

9. The system of claim 1 , wherein generating the object hierarchy comprises:

obtaining a first hierarchical representation of the first plurality of active GUI elements of the first application program from an operating system; and

identifying the first plurality of active GUI elements of the first application program using the first hierarchical representation.

10. The system of claim 9 , wherein obtaining the first hierarchical representation is performed by using the WINDOWS Automation API.

11. The system of claim 1 , wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform:

controlling the first application program to perform a third action in the sequence of actions using the first refreshed object hierarchy at least in part by:

using a third object of the third plurality of objects to cause, via a third active GUI element of the third plurality of active GUI elements, the first application program to at least partially perform the third action, the third action programmatically emulating a user's interaction with the third active GUI element.

12. The system of claim 1 , wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform:

controlling the second application program to perform a fourth action in the sequence of actions using the second refreshed object hierarchy at least in part by:

using a fourth object of the fourth plurality of objects to cause, via a fourth active GUI element of the fourth plurality of active GUI elements, the second application program to at least partially perform the fourth action, the fourth action programmatically emulating a user's interaction with the fourth active GUI element.

13. The system of claim 1 , wherein the first and second application programs are configured to execute using different operating systems.

14. The system of claim 1 , wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

15. The system of claim 1 , wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program.

16. The system of claim 1 , wherein the first and second application programs are configured to execute at least in part by using different software platforms.

17. The system of claim 16 , wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, and an ORACLE platform.

18. A method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising:

using at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising:

identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;

generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by:

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element; and

after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program; and

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by:

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element; and

after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.

19. The method of claim 18 , wherein refreshing the object hierarchy comprises:

refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one object from the object hierarchy.

20. The method of claim 18 , wherein refreshing the object hierarchy comprises: refreshing only a subset of objects in the object hierarchy.

21. The method of claim 18 , wherein controlling the first application program to perform the first action in the sequence of actions comprises:

accessing the first object in the object hierarchy corresponding to the first active GUI element of the first application program.

22. The method of claim 21 , wherein refreshing the object hierarchy comprises:

refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object.

23. The method of claim 18 , wherein generating the object hierarchy comprises:

obtaining a first hierarchical representation of the first plurality of active GUI elements of the first application program from an operating system; and

identifying the first plurality of active GUI elements of the first application program using the first hierarchical representation.

24. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, causes the at least one computer hardware processor to perform a method for controlling at least one application program for performing a task comprising a sequence of actions, the at least one application program including a first application program and a second application program, the first and second application programs configured to execute at least in part by using different graphical user interface (GUI) application libraries, the method comprising:

generating an object hierarchy comprising a plurality of objects corresponding to active GUI elements of the at least one application program, the generating comprising:

identifying a first plurality of active GUI elements of the first application program and a second plurality of active GUI elements of the second application program, wherein the first application program is configured to execute by using a first GUI application library and the second application program is configured to execute by using a second GUI application library that is different than the first GUI application library;

generating a first portion of the object hierarchy comprising a first plurality of objects corresponding to the first plurality of active GUI elements of the first application program and a second portion of the object hierarchy comprising a second plurality of objects corresponding to the second plurality of active GUI elements of the second application program;

controlling the first application program to perform a first action in the sequence of actions using the first portion of the object hierarchy at least in part by:

using a first object of the first plurality of objects to cause, via a first active GUI element of the first plurality of active GUI elements, the first application program to at least partially perform the first action, the first action programmatically emulating a user's interaction with the first active GUI element; and

after the first application program performs the first action, refreshing the object hierarchy to obtain a first refreshed object hierarchy comprising a third plurality of objects corresponding to a third plurality of active GUI elements reflecting changes in a user interface of the first application program;

controlling the second application program to perform a second action in the sequence of actions using the second portion of the object hierarchy at least in part by:

using a second object of the second plurality of objects to cause, via a second active GUI element of the second plurality of active GUI elements, the second application program to at least partially perform the second action, the second action programmatically emulating a user's interaction with the second active GUI element; and

after the second application program performs the second action, refreshing the object hierarchy to obtain a second refreshed object hierarchy comprising a fourth plurality of objects corresponding to a fourth plurality of active GUI elements reflecting changes in a user interface of the second application program.

25. The at least one non-transitory computer-readable storage medium of claim 24 , wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.