User interface (UI) automation troubleshooting systems and methods
In some embodiments, a troubleshooting module executes on a host machine alongside an instance of a robotic process automation (RPA) robot that carries out an automation workflow. The troubleshooting module is configured to detect an automation failure and in response, to carry out a forensic investigation to determine a cause of the malfunction. Exemplary causes detected by the troubleshooter include focus theft and an inadvertent file lock. The troubleshooting module is further configured to identify a culprit (i.e., software entity causing the respective failure) and to communicate a descriptor of the culprit to a user, for instance by way of a failure notification displayed on a screen of the respective host machine or within an administration interface exposed by an RPA conductor orchestrating the execution of multiple RPA robots.
1 . A computer system comprising at least one hardware processor configured to execute a troubleshooter module communicatively coupled to a robotic process automation (RPA) robot configured to execute an RPA activity mimicking a human interaction with a user interface (UI) exposed by the computer system, the troubleshooter module configured to troubleshoot the RPA activity, comprising:
in response to a failure of the RPA robot to execute the RPA activity, determining whether the failure was caused by UI focus theft according to a content of a data structure representing the UI in a memory of the computer system;
in response to determining that the failure was caused by UI focus theft, identifying a culprit software entity causing the UI focus theft; and
initiating a display of a failure notification to a user, the failure notification including an indicator of a cause of the failure and an identifier of the culprit software entity.
2 . The computer system of claim 1 , wherein determining whether the failure was caused by UI focus theft comprises identifying within the UI a window currently in focus.
3 . The computer system of claim 2 , wherein the RPA activity comprises acting on an element displayed within a target window of the UI, and wherein determining whether the failure was caused by UI focus theft comprises determining whether the window currently in focus coincides with the target window.
4 . The computer system of claim 2 , wherein identifying the window currently in focus comprises:
transmitting a screenshot showing at least a part of the UI to an image-processing artificial intelligence (AI) module pre-trained to detect UI windows present within the screenshot; and
receiving from the AI module a descriptor of the window currently in focus.
5 . The computer system of claim 1 , wherein determining whether the failure was caused by UI focus theft comprises identifying within the UI a window having focus at a time of the failure.
6 . The computer system of claim 5 , wherein:
the troubleshooter module is configured to construct an event log comprising a time record of UI focus change events having occurred within the UI, the event log comprising, for each UI focus change event, a time of occurrence of the respective event and an identifier of a window receiving focus in response to the respective event; and
identifying the window having focus at the time of the failure comprises:
determining the time of the failure according to an output of the RPA robot, and
parsing the event log according to the time of the failure.
7 . The computer system of claim 1 , wherein:
the failure notification is displayed within a user interface of an RPA conductor orchestrating the execution of multiple RPA robots including the RPA robot; and
initiating the display of the failure notification comprises transmitting a troubleshooting report to the RPA conductor, the troubleshooting report including the indicator of the cause of the failure and the identifier of the culprit software entity.
8 . The computer system of claim 1 , wherein:
the RPA activity comprises acting on a UI element displayed within a target window of the UI;
the troubleshooter module is further configured to identify a focus stealing window comprising a window of the UI stealing focus from the target window; and
the failure notification is further indicative of a content of the focus stealing window.
9 . The computer system of claim 8 , wherein the target window comprises a web browser window displaying a target web page and wherein the focus stealing window comprises an overlay obscuring at least a part of the target web page.
10 . The computer system of claim 8 , wherein the content of the focus stealing window comprises an item selected from a group consisting of an error message, a software update notification, a calendar notification, and a notification of an incoming message.
11 . A robotic process automation (RPA) troubleshooting method comprising employing at least one hardware processor of a computer system to execute a troubleshooter module communicatively coupled to an RPA robot configured to execute an RPA activity mimicking a human interaction with a user interface (UI) exposed by the computer system, wherein executing the troubleshooter module comprises:
in response to a failure of the RPA robot to execute the RPA activity, determining whether the failure was caused by UI focus theft according to a content of a data structure representing the UI in a memory of the computer system;
in response to determining that the failure was caused by UI focus theft, identifying a culprit software entity causing the UI focus theft; and
initiating a display of a failure notification to a user, the failure notification including an indicator of a cause of the failure and an identifier of the culprit software entity.
12 . The method of claim 11 , wherein determining whether the failure was caused by UI focus theft comprises identifying within the UI a window currently in focus.
13 . The method of claim 12 , wherein the RPA activity comprises acting on an element displayed within a target window of the UI, and wherein determining whether the failure was caused by UI focus theft comprises determining whether the window currently in focus coincides with the target window.
14 . The method of claim 12 , wherein identifying the window currently in focus comprises:
transmitting a screenshot showing at least a part of the UI to an image-processing artificial intelligence (AI) module pre-trained to detect UI windows present within the screenshot; and
receiving from the AI module a descriptor of the window currently in focus.
15 . The method of claim 11 , wherein determining whether the failure was caused by UI focus theft comprises identifying within the UI a window having focus at a time of the failure.
16 . The method of claim 15 , wherein:
the troubleshooter module is configured to construct an event log comprising a time record of UI focus change events having occurred within the UI, the event log comprising, for each UI focus change event, a time of occurrence of the respective event and an identifier of a window receiving focus in response to the respective event; and
identifying the window having focus at the time of the failure comprises:
determining the time of the failure according to an output of the RPA robot, and
parsing the event log according to the time of the failure.
17 . The method of claim 11 , wherein:
the failure notification is displayed within a user interface of an RPA conductor orchestrating the execution of multiple RPA robots including the RPA robot; and
initiating the display of the failure notification comprises transmitting a troubleshooting report to the RPA conductor, the troubleshooting report including the indicator of the cause of the failure and the identifier of the focus thief.
18 . The method of claim 11 , wherein:
the RPA activity comprises acting on a UI element displayed within a target window of the UI;
the troubleshooter module is further configured to identify a focus stealing window comprising a window of the UI stealing focus from the target window; and
the failure notification is further indicative of a content of the focus stealing window.
19 . The method of claim 11 , wherein the target window comprises a web browser window displaying a target web page and wherein the focus stealing window comprises an overlay obscuring at least a part of the target web page.
20 . The method of claim 11 , wherein the content of the focus stealing window comprises an item selected from a group consisting of an error message, a software update notification, a calendar notification, and a notification of an incoming message.
21 . A non-transitory computer-readable medium storing instructions which, when executed by at least one hardware processor of a computer system, cause the computer system to execute a troubleshooter module communicatively coupled to a robotic process automation (RPA) robot configured to execute an RPA activity mimicking a human interaction with a user interface (UI) exposed by the computer system, the troubleshooter module configured to troubleshoot the RPA activity, comprising:
in response to a failure of the RPA robot to execute the RPA activity, determining whether the failure was caused by UI focus theft according to a content of a data structure representing the UI in a memory of the computer system;
in response to determining that the failure was caused by UI focus theft, identifying a culprit software entity causing the UI focus theft; and
initiating a display of a failure notification to a user, the failure notification including an indicator of a cause of the failure and an identifier of the culprit software entity.