Graphical element search technique selection, fuzzy logic selection of anchors and targets, and/or hierarchical graphical element identification for robotic process automation
Graphical element search technique selection, fuzzy logic selection for anchors and targets, and hierarchical graphical element identification for robotic process automation (RPA) are disclosed. The fuzzy logic selection of anchors and targets may be part of a larger, tiered, or hierarchical process for identifying graphical elements in the UI. When a selector for a UI element is not found with at least a confidence threshold, similar elements potentially corresponding to the selector for a UI element target may be searched based on fuzzy matching of the target and corresponding anchor(s). Geometric matching may also be employed between the target UI element and its respective anchor(s). The combination of fuzzy matching and geometric matching may allow for more flexible and accurate identification of the exact selector with which an RPA robot is attempting to interact.
1 . A computer-implemented method, comprising:
designating a target user interface (UI) element in an image of a portion or all of a Ul of an application or of a UI of a live application on which automation is to be performed at design time, by a robotic process automation (RPA) designer application, wherein the designating comprises a user selecting the UI element directly on the Ul of the application or the Ul of the live application displayed on a display device;
receiving a selection of one or more search algorithms of a plurality of search algorithms at design time, by the RPA designer application via a selection options pane, wherein the one or more search algorithms comprise at least one fuzzy selector; and
configuring an activity of an RPA workflow for an RPA robot to execute the selected one or more search algorithms comprising the at least one fuzzy selector at design time, by the RPA designer application, wherein
an accuracy of the at least one fuzzy selector is configurable via the selection options pane.
2 . The computer-implemented method of claim 1 , wherein the one or more search algorithms additionally comprise a selector search, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof.
3 . The computer-implemented method of claim 2 , wherein the one or more search algorithms comprise parsing a UI tree for the UI of the application to identify one or more attributes of the target Ul element, one or more anchors, one or more tags, or a combination thereof.
4 . The computer-implemented method of claim 3 , wherein all tags in the Ul tree are found in one parse.
5 . The computer-implemented method of claim 2 , wherein
the one or more selected search algorithms comprise text matching, and
the text matching requires an exact match for text of candidates for the target Ul element.
6 . The computer-implemented method of claim 2 , wherein
the one or more selected search algorithms comprise image matching, and
a match of an image associated with the target Ul element is mandatory for the image matching to find a match.
7 . The computer-implemented method of claim 2 , wherein
the at least one fuzzy selector is configured to perform image matching.
8 . The computer-implemented method of claim 1 , wherein the configuring of the RPA workflow activity comprises configuring one or more anchors, one or more tag attributes, one or more accuracy levels, or a combination thereof.
9 . The computer-implemented method of claim 1 , wherein the one or more selected search algorithms in the activity are configured to be executed in an order, and
execution is stopped when a currently executing search algorithm finds the target UI element.
10 . The computer-implemented method of claim 1 , further comprising:
executing a selector search for the target UI element at runtime to identify the target UI element, by the RPA robot implementing the RPA workflow;
responsive to the target UI element not being found, executing a fuzzy selector search via the at least one fuzzy selector, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, to identify the target UI element, by the RPA robot; and
performing an interaction with the identified target UI element, by the RPA robot, based on the configured activity.
11 . The computer-implemented method of claim 10 , wherein
two or more of the fuzzy selector search, the target and anchor search, the fuzzy target and anchor search, the image matching, the text matching, and the computer vision (CV) matching are executed, and
the execution of the two or more of the fuzzy selector search, the target and anchor search, the fuzzy target and anchor search, the image matching, the text matching, and the CV matching is performed in an order in the configured activity.
12 . The computer-implemented method of claim 10 , wherein
the fuzzy target and anchor search is performed by the RPA robot, and
the fuzzy target and anchor search comprises determining the target and at least one anchor using fuzzy matching and performing geometric matching on the target and the at least one anchor determined using the fuzzy matching.
13 . The computer-implemented method of claim 12 , wherein the fuzzy target and anchor search comprises calling a driver,
a single driver call is made for the target and the at least one anchor, and
the driver obtains all candidates for the target and the at least one anchor in a single traversal of a UI tree for the application associated with the UI.
14 . The computer-implemented method of claim 1 , further comprising:
executing a selector search and at least one of a fuzzy selector search via the at least one fuzzy selector, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, and computer vision (CV) matching for the target UI element at runtime, by the RPA robot configured to implement the RPA workflow;
selecting a best result from the selector search and the at least one of the fuzzy selector search, the target and anchor search, the fuzzy target and anchor search, the image matching, the text matching, and the CV matching based on validation scores, by the RPA robot, to identify the target Ul element; and
performing an interaction with the identified target Ul element, by the RPA robot, based on the configured activity.
15 . A non-transitory computer-readable medium storing a computer program for a robotic process automation (RPA) designer application, the computer program configured to cause at least one processor to:
designate a target user interface (UI) element in an image of a portion or all of a UI of an application or of a UI of a live application on which automation is to be performed at design time, wherein the designating comprises a user selecting the UI element directly on the UI of the application or the Ul of the live application displayed on a display device;
receive a selection of one or more search algorithms of a plurality of search algorithms at design time via a selection options pane, wherein the one or more search algorithms comprise at least one fuzzy selector; and
configure an activity of an RPA workflow te for an RPA robot to execute the selected one or more search algorithms at design time, wherein
the one or more search algorithms comprise a selector search, a fuzzy selector search performed via the at least one fuzzy selector, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, and
an accuracy of the at least one fuzzy selector is configurable via the selection options pane.
16 . The non-transitory computer-readable medium of claim 15 , wherein
the one or more search algorithms comprise parsing a UI tree for the UI of the application to identify one or more attributes of the target UI element, one or more anchors, one or more tags, or a combination thereof, and
all tags in the UI tree are found in one parse.
17 . The non-transitory computer-readable medium of claim 15 , wherein
the one or more selected search algorithms comprise text matching, and
the text matching requires an exact match for text of candidates for the target UI element.
18 . The non-transitory computer-readable medium of claim 15 , wherein
the one or more selected search algorithms comprise image matching, and
a match of an image associated with the target UI element is mandatory for the image matching to find a match.
19 . The non-transitory computer-readable medium of claim 15 , wherein
the at least one fuzzy selector is configured to perform image matching.
20 . The non-transitory computer-readable medium of claim 15 , wherein the configuring of the RPA workflow activity comprises configuring one or more anchors, one or more tag attributes, one or more accuracy levels, or a combination thereof.
21 . The non-transitory computer-readable medium of claim 15 , wherein
the one or more selected search algorithms in the activity are configured to be executed in an order, and
execution is stopped when a currently executing search algorithm finds the target UI element.
22 . A computing system, comprising:
memory storing computer program instructions; and
at least one processor configured to execute the computer program instructions, wherein the computer program instructions are configured to cause at least one processor to:
designate a target user interface (UI) element in an image of a portion or all of a UI of an application or of a UI of a live application on which automation is to be performed at design time, wherein the designating comprises a user selecting the UI element directly on the UI of the application or the UI of the live application displayed on a display device;
receive a selection of one or more search algorithms of a plurality of search algorithms at design time via a selection options pane, wherein the one or more search algorithms comprise at least one fuzzy selector; and
configure an activity of an RPA workflow to for an RPA robot to execute the selected one or more search algorithms comprising the at least one fuzzy selector at design time, wherein
the one or more search algorithms comprise a selector search, a fuzzy selector search performed via the at least one fuzzy selector, a target and anchor search, a fuzzy target and anchor search, image matching, text matching, computer vision (CV) matching, or a combination thereof, and
an accuracy of the at least one fuzzy selector is configurable via the selection options pane.
23 . The computing system of claim 22 , wherein the computer program instructions comprise a computer program for a robotic process automation (RPA) designer application.
24 . The computing system of claim 22 , wherein
the one or more search algorithms comprise parsing a UI tree for the UI of the application to identify one or more attributes of the target Ul element, one or more anchors, one or more tags, or a combination thereof, and
all tags in the Ul tree are found in one parse.
25 . The computing system of claim 22 , wherein
the one or more selected search algorithms comprise text matching, and the text matching requires an exact match for text of candidates for the target UI element.
26 . The computing system of claim 22 , wherein
the one or more selected search algorithms comprise image matching, and a match of an image associated with the target UI element is mandatory for the image matching to find a match.
27 . The computing system of claim 22 , wherein
the at least one fuzzy selector is configured to perform image matching.
28 . The computing system of claim 22 , wherein the configuring of the RPA workflow activity comprises configuring one or more anchors, one or more tag attributes, one or more accuracy levels, or a combination thereof.
29 . The computing system of claim 22 , wherein
the one or more selected search algorithms in the activity are configured to be executed in an order, and
execution is stopped when a currently executing search algorithm finds the target UI element.