Graphical element search technique selection, fuzzy logic selection of anchors and targets, and/or hierarchical graphical element identification for robotic process automation
Graphical element search technique selection, fuzzy logic selection for anchors and targets, and hierarchical graphical element identification for robotic process automation (RPA) are disclosed. The fuzzy logic selection of anchors and targets may be part of a larger, tiered, or hierarchical process for identifying graphical elements in the UI. When a selector for a UI element is not found with at least a confidence threshold, similar elements potentially corresponding to the selector for a UI element target may be searched based on fuzzy matching of the target and corresponding anchor(s). Geometric matching may also be employed between the target UI element and its respective anchor(s). The combination of fuzzy matching and geometric matching may allow for more flexible and accurate identification of the exact selector with which an RPA robot is attempting to interact.
1 . A computer-implemented method for identifying a visible target user interface (UI) element associated with an activity of a robotic process automation (RPA) workflow, comprising:
executing a selector search for the target UI element at runtime to identify the visible target UI element, by an RPA robot implementing the RPA workflow;
responsive to the visible target UI element not being found, executing at least two of a fuzzy selector search, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, in parallel to identify the target UI element, by the RPA robot; and
performing an interaction with the identified target UI element, by the RPA robot, based on the activity of the RPA workflow.
2 . The computer-implemented method of claim 1 , wherein
the execution of the at least two of the fuzzy selector search, the target and anchor search, the fuzzy target and anchor search, the image matching, the text matching, and the CV matching are performed based on an order in the activity of the RPA workflow.
3 . The computer-implemented method of claim 1 , wherein
the fuzzy target and anchor search is performed by the RPA robot, and
the fuzzy target and anchor search comprises determining the target and at least one anchor using fuzzy matching and performing geometric matching on the target and the at least one anchor determined using the fuzzy matching.
4 . The computer-implemented method of claim 3 , wherein
the fuzzy target and anchor search comprises calling a driver,
a single driver call is made for the target and the at least one anchor, and
the driver obtains all candidates for the target and the at least one anchor in a single traversal of a UI a tree for an application associated with the UI.
5 . A computer-implemented method for identifying a target user interface (UI) element associated with an activity of a robotic process automation (RPA) workflow, comprising:
executing in parallel a selector search and at least two of a fuzzy selector search, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, and computer vision (CV) matching for the target UI element at runtime, by an RPA robot configured to implement the RPA workflow;
selecting a best result from the selector search and the at least one of the fuzzy selector search, the target and anchor search, the fuzzy target and anchor search, the image matching, the text matching, and the CV matching based on validation scores, by the RPA robot, to identify the target UI element; and
performing an interaction with the identified target UI element, by the RPA robot, based on the activity of the RPA workflow.
6 . The computer-implemented method of claim 5 , wherein
the fuzzy target and anchor search is performed by the RPA robot, and
the fuzzy target and anchor search comprises determining the target and at least one anchor using fuzzy matching and performing geometric matching on the target and the at least one anchor determined using the fuzzy matching.
7 . The computer-implemented method of claim 1 , wherein after performing the interaction with the identified target UI element based on the activity of the RPA workflow, the method further comprises:
proceeding to a next activity in the RPA workflow that requires identification of another target UI element; and
repeating the method of claim 1 for the other target UI element.
8 . The computer-implemented method of claim 1 , wherein a UI action of the activity comprises clicking a button, entering text in a text field, selecting a menu item, opening a window, closing the window, or submitting a form.
9 . The computer-implemented method of claim 8 , wherein
the fuzzy target and anchor search comprises calling a driver,
a single driver call is made for the target and the at least one anchor, and
the driver obtains all candidates for the target and the at least one anchor in a single traversal of a UI a tree for an application associated with the UI.
10 . The computer-implemented method of claim 5 , wherein after performing the interaction with the identified target UI element based on the activity of the RPA workflow, the method further comprises:
proceeding to a next activity in the RPA workflow that requires identification of another target UI element; and
repeating the method of claim 5 for the other target UI element.
11 . The computer-implemented method of claim 5 , wherein a UI action of the activity comprises clicking a button, entering text in a text field, selecting a menu item, opening a window, closing the window, or submitting a form.
12 . A non-transitory computer-readable medium storing a computer program for identifying a target user interface (UI) element associated with an activity of a robotic process automation (RPA) workflow, the computer program configured to cause at least one processor to:
execute a selector search for the target UI element at runtime to identify the target UI element, and responsive to the target UI element not being found, execute at least one other search algorithm to identify the target UI element and perform an interaction with the identified target UI element based on the activity of the RPA workflow; or
execute the selector search and the at least one other search algorithm, select a best result from the selector search and the at least one other search algorithm based on validation scores to identify the target UI element, and perform the interaction with the identified target UI element based on the activity of the RPA workflow, wherein
the at least one other search algorithm comprises a fuzzy selector search, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, wherein
the at least one other search algorithm comprises at least two search algorithms, and
the at least two search algorithms are performed in parallel.
13 . The non-transitory computer-readable medium of claim 12 , wherein two or more of the fuzzy selector search, the target and anchor search, the fuzzy target and anchor search, the image matching, the text matching, and the CV matching are performed based on an order in the activity of the RPA workflow.
14 . The non-transitory computer-readable medium of claim 12 , wherein the fuzzy target and anchor search comprises determining the target and at least one anchor using fuzzy matching and performing geometric matching on the target and the at least one anchor determined using the fuzzy matching.
15 . The non-transitory computer-readable medium of claim 14 , wherein
the fuzzy target and anchor search comprises calling a driver,
a single driver call is made for the target and the at least one anchor, and
the driver obtains all candidates for the target and the at least one anchor in a single traversal of a UI a tree for an application associated with the UI.
16 . The non-transitory computer-readable medium of claim 12 , wherein after performing the interaction with the identified target UI element based on the activity of the RPA workflow, the method further comprises:
proceeding to a next activity in the RPA workflow that requires identification of another target UI element; and
repeating the steps of claim 12 for the other target UI element.
17 . The non-transitory computer-readable medium of claim 16 , wherein responsive to all activities in the RPA workflow having been completed, the computer program ends.
18 . The non-transitory computer-readable medium of claim 12 , wherein a UI action of the activity comprises clicking a button, entering text in a text field, selecting a menu item, opening a window, closing the window, or submitting a form.
19 . The non-transitory computer-readable medium of claim 12 , wherein the computer program is an RPA robot.