Detecting user interface elements in robotic process automation using convolutional neural networks
Graphical elements in a user interface (UI) may be detected in robotic process automation (RPA) using convolutional neural networks (CNNs). Such processes may be particularly well-suited for detecting graphical elements that are too small to be detected using conventional techniques. The accuracy of detecting graphical elements (e.g., control objects) may be enhanced by providing neural network-based processing that is robust to changes in various UI factors, such as different resolutions, different operating system (OS) scaling factors, different dots-per-inch (DPI) settings, and changes due to UI customization of applications and websites, for example.
1. A computer program embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:
create a raw dataset by collecting images directly from an environment on which a convolutional neural network (CNN) will operate;
augment the raw dataset using random cropping to produce an augmented dataset comprising cropped images;
overlay rectangles over a cropped image of the produced cropped images as a grid;
match the overlaid rectangles to actual labels;
when a match occurs, use the cropped image as a positive example to train the CNN;
when a match does not occur, use the cropped image as a negative example to train the CNN;
train the CNN using the augmented dataset; and
detect graphical elements in a user interface (UI) using the trained CNN.
2. The computer program of claim 1 , wherein the raw dataset is created from synthetic data mimicking real screenshots.
3. The computer program of claim 1 , wherein the augmenting of the raw dataset comprises flipping images horizontally, changing colors, artificially adding noise, artificially adding lighting, or any combination thereof.
4. The computer program of claim 1 , wherein the augmenting of the raw data further comprises using channel randomization, hue shift, color inversion, random scaling, blurring of images, or any combination thereof.
5. The computer program of claim 1 , wherein
the detecting of the graphical elements comprises providing a fixed number of proposals for each graphical element, and
the proposals are filtered with different thresholds for design time and for runtime, the runtime threshold having a higher detection probability than the design time threshold.
6. The computer program of claim 1 , wherein the program is further configured to cause the at least one processor to:
provide proposals for each rectangle in the grid, wherein
the proposals comprise a scale and a stride distance, the stride distance defining a distance between the rectangles, wherein
each time two rectangles are compared, an intersection over a union or an intersection over a minimum is used with a given adaptive threshold that depends on an area of a given rectangle.
7. The computer program of claim 1 , wherein different cropping techniques are used to identify at least two different graphical element types.
8. The computer program of claim 1 , wherein the CNN comprises a Faster Region-based Convolutional Neural Network (R-CNN) architecture with a region proposal network (RPN).
9. The computer program of claim 1 , wherein the augmented dataset comprises translations.
10. A computer-implemented method, comprising:
augmenting, by a computing system, a raw dataset using random cropping to produce an augmented dataset comprising cropped images;
overlaying rectangles over a cropped image of the produced cropped images as a grid, by the computing system;
matching the overlaid rectangles to actual labels, by the computing system;
providing proposals for each rectangle in the grid, by the computing system;
when a match occurs, using the cropped image as a positive example to train the CNN, by the computing system;
when a match does not occur, using the cropped image as a negative example to train the CNN, by the computing system
training a convolutional neural network (CNN), by the computing system, using the augmented dataset; and
detecting graphical elements in a user interface (UI), by the computing system, using the trained CNN.
11. The computer-implemented method of claim 10 , wherein
the detecting of the graphical elements comprises providing a fixed number of proposals for each graphical element, and
the proposals are filtered with different thresholds for design time and for runtime, the runtime threshold having a higher detection probability than the design time threshold.
12. The computer-implemented method of claim 10 , wherein the augmenting of the raw data further comprises using channel randomization, hue shift, color inversion, random scaling, blurring of images, or any combination thereof.
13. The computer-implemented method of claim 10 , wherein different cropping techniques are used to identify at least two different graphical element types.
14. The computer-implemented method of claim 10 , wherein
each time two rectangles are compared, an intersection over a union or an intersection over a minimum is used with a given threshold, and
the threshold is an adaptive threshold that depends on an area of a given rectangle.
15. The computer-implemented method of claim 10 , further comprising:
creating the raw dataset, by the computing system, by collecting images directly from an environment on which the CNN will operate, wherein
the raw dataset is created from synthetic data mimicking real screenshots, and
the augmenting of the raw dataset comprises flipping images horizontally, changing colors, artificially adding noise, artificially adding lighting, or any combination thereof.
16. A system, comprising:
memory storing computer program instructions; and
at least one processor configured to execute the computer program instructions, the at least one processor configured to:
detect graphical elements in a user interface (UI) using a Faster Region-based Convolutional Neural Network (R-CNN) architecture with a region proposal network (RPN) backbone, the detection comprising overlaying rectangles over an image as a grid and providing a predetermined number of proposals for each rectangle in the grid, wherein
the proposals comprise a scale and a stride distance, the stride distance defining a distance between the rectangles,
each time two rectangles are compared, an intersection over a union or an intersection over a minimum is used with a given threshold, and
the threshold is an adaptive threshold that depends on an area of a given rectangle.
17. The system of claim 16 , wherein the proposals are filtered with different thresholds for design time and for runtime, the runtime threshold having a higher detection probability than the design time threshold.
18. The system of claim 16 , wherein the at least one processor is further configured to:
create a raw dataset by collecting images directly from an environment on which the Faster R-CNN will operate; and
augment the raw dataset to produce an augmented dataset, wherein
the raw dataset is created from synthetic data mimicking real screenshots, and
the augmenting of the raw dataset comprises flipping images horizontally, changing colors, artificially adding noise, artificially adding lighting, or any combination thereof.
19. The system of claim 18 , wherein
the augmenting of the raw data comprises using channel randomization, hue shift, color inversion, random cropping, random scaling, blurring of images, or any combination thereof, and
random cropping is used to produce cropped images and the at least one processor is further configured to:
overlay rectangles over a cropped image of the produced cropped images as a grid;
match the overlaid rectangles to actual labels;
when a match occurs, use the cropped image as a positive example to train the CNN; and
when a match does not occur, use the cropped image as a negative example to train the CNN.