Software robots for programmatically controlling computer programs to perform tasks

1. A system, comprising:

at least one hardware processor; and

at least one non-transitory computer-readable storage medium storing:

a computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and

processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising:

identifying a first plurality of active GUI elements associated with a user interface of the first application program; and

generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements; and

controlling the first application program to perform the first sub-task at least in part by:

identifying, using the computer program, a first action to perform in furtherance of the first sub-task;

accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;

using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;

identifying, using the computer program, a second action to perform in furtherance of the first sub-task;

refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements; and

using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.

2. The system of claim 1 , wherein the task further comprises a second sub-task to be performed by a second application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the second application program to perform the second sub-task at least in part by:

identifying, using the computer program, a third action to perform in furtherance of the second sub-task;

automatically accessing, in the object hierarchy, a third object corresponding to a third active GUI element of the second application program; and

automatically using the third object to cause, via the third active GUI element, the second application program to at least partially perform the third action.

3. The system of claim 2 , wherein the task further comprises a third sub-task to be performed by a third application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the third application program to perform the third sub-task at least in part by:

identifying, using the computer program, a fourth action to perform in furtherance of the third sub-task;

automatically accessing, in the object hierarchy, a fourth object corresponding to a fourth active GUI element of the third application program; and

automatically using the fourth object to cause, via the fourth active GUI element, the third application program to at least partially perform the fourth action.

4. The system of claim 1 , wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.

5. The system of claim 1 , wherein refreshing the object hierarchy comprises refreshing only the first object and/or one or more descendant objects of the first object in the object hierarchy.

6. The system of claim 1 , wherein refreshing the object hierarchy comprises adding to the object hierarchy a new object associated with an active GUI element of the first application program, wherein the second object comprises the new object.

7. The system of claim 1 , wherein refreshing the object hierarchy further comprises removing a particular object from the object hierarchy, wherein the particular object is associated with an inactive GUI element of the first application program.

8. The system of claim 1 , wherein refreshing the object hierarchy comprises updating at least one property of at least one object in the object hierarchy.

9. The system of claim 1 , wherein refreshing the object hierarchy comprises:

determining that a particular object in the object hierarchy is invalid; and

refreshing the particular object using a reverse tree traversal technique.

10. The system of claim 1 , wherein the accessing comprises searching for the first object in the object hierarchy using a search string constructed in accordance with a search grammar, the search grammar comprising at least one token operator, at least one action operator, and at least one relationship operator.

11. The system of claim 1 , wherein refreshing the object hierarchy comprises:

refreshing one or more objects in the object hierarchy, wherein each of the one or more objects is a descendant of the first object, and wherein refreshing the one or more objects is performed using a plurality of threads.

12. The system of claim 11 , wherein the one or more objects includes a first child object of the first object and a second child object of the first object, and wherein refreshing the one or more objects comprises:

refreshing objects in the object hierarchy that are descendants of the first child object using a first thread in the plurality of threads; and

refreshing objects in the object hierarchy that are descendants of the second child object using a second thread in the plurality of threads.

13. The system of claim 12 , wherein the object hierarchy includes a first set of objects associated with the first application program and a second set of objects associated with a second application program, wherein the first set of objects includes more objects than that of the second set of objects, and wherein refreshing objects in the object hierarchy using the plurality of threads comprises using more threads in the plurality of threads to refresh objects in the first set of objects than that to refresh objects in the second set of objects.

14. The system of claim 1 , wherein refreshing the object hierarchy comprises refreshing an object in the object hierarchy, and the second object comprises the refreshed object.

15. The system of claim 1 , wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

16. The system of claim 1 , wherein the first active GUI element of the first application program is a hidden GUI element of the first application program.

17. A method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

using at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising:

identifying a first plurality of active GUI elements associated with a user interface of the first application program; and

generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements; and

controlling the first application program to perform the first sub-task at least in part by:

identifying, using a computer program, a first action to perform in furtherance of the first sub-task;

accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;

using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;

identifying, using the computer program, a second action to perform in furtherance of the first sub-task;

refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements; and

using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.

18. The method of claim 17 , wherein refreshing the object hierarchy comprises adding to the object hierarchy a new object associated with an active GUI element of the first application program, and the second object comprises the new object.

19. The method of claim 17 , wherein refreshing the object hierarchy comprises refreshing an object in the object hierarchy, and the second object comprises the refreshed object.

20. A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one hardware processor, cause the at least one hardware processor to perform a method for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program, the method comprising:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program, the generating comprising:

identifying a first plurality of active GUI elements associated with a user interface of the first application program; and

generating, as part of the object hierarchy, a first plurality of objects corresponding to the first plurality of active GUI elements; and

controlling the first application program to perform the first sub-task at least in part by:

identifying, using a computer program, a first action to perform in furtherance of the first sub-task;

accessing, in the object hierarchy, a first object of the first plurality of objects corresponding to a first active GUI element of the first plurality of active GUI elements of the first application program;

using the first object to cause, via the first active GUI element, the first application program to at least partially perform the first action;

identifying, using the computer program, a second action to perform in furtherance of the first sub-task;

refreshing the object hierarchy to obtain a refreshed object hierarchy comprising a second plurality of objects corresponding to a second plurality of active GUI elements associated with the user interface of the first application program, the second plurality of objects including a second object not in the first plurality of objects, the second object corresponding to a second active GUI element of the second plurality of active GUI elements; and

using the second object to cause, via the second active GUI element, the first application program to at least partially perform the second action.