Self-learning robotic process automation
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a self-learning robotic process automation are disclosed. In one aspect, a method includes receiving an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface. The method further includes executing a command of the one or more commands of the automated script. The method further includes determining that an error occurred during execution of the command of the one or more commands of the automated script. The method further includes determining a modification for the command by applying a script repair model. The method further includes executing the modified command. The method further includes determining whether the error or another error occurred during execution of the modified command. The method further includes determining whether to update the automated script with the modified command.
1. A computer-implemented method comprising:
providing, for execution and by the computing device, a command of an automated script that is configured to (i) use computer vision to identify, in a graphical interface, a first group of pixels that match an image and (ii) interact with the first group of pixels;
determining, by the computing device, that an error occurred during execution of the command of the automated script;
in response to determining that the error occurred during execution of the command of the automated script, modifying, by the computing device, the command of the automated script by adjusting the image;
providing, for execution and by the computing device, the adjusted command of the automated script that is configured to (i) use computer vision to identify, in the graphical interface, a second group of pixels that match the adjusted image and (ii) interact with the second group of pixels;
determining, by the computing device, that the error or another error did not occur during execution of the automated script; and
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device, the command of the automated script using the adjusted image.
2. The method of claim 1 , wherein modifying the command of the automated script by adjusting the image comprises cropping the image.
3. The method of claim 1 , comprising:
determining that the graphical interface does not include the first group of pixels that match the image,
wherein modifying the command of the automated script is based on determining that the graphical interface does not include the first group of pixels that match the image.
4. The method of claim 1 , wherein determining, by the computing device, that an error occurred during execution of the command of the automated script comprises:
determining that the graphical interface does not include the first group of pixels.
5. The method of claim 1 , comprising:
in response to determining that the error occurred during execution of the command of the automated script, providing, by the computing device, the image and a representation of the graphical interface as an input to a script repair model that is configured to determine an adjustment to the image; and
receiving, from the script repair model, data indicating the adjustment to the image,
wherein adjusting the image is based on the data indicating the adjustment to the image.
6. The method of claim 5 , comprising:
receiving, by the computing device, one or more additional automated scripts that each include commands and results that correspond to each command; and
training, using machine learning, the script repair model using the one or more additional automated scripts that each include commands and the results that correspond to each command.
7. The method of claim 5 , comprising:
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device and using machine learning, the script repair model using the adjusted command and the adjusted image.
8. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
providing, for execution and by the computing device, a command of an automated script that is configured to (i) use computer vision to identify, in a graphical interface, a first group of pixels that match an image and (ii) interact with the first group of pixels;
determining, by the computing device, that an error occurred during execution of the command of the automated script;
in response to determining that the error occurred during execution of the command of the automated script, modifying, by the computing device, the command of the automated script by adjusting the image;
providing, for execution and by the computing device, the adjusted command of the automated script that is configured to (i) use computer vision to identify, in the graphical interface, a second group of pixels that match the adjusted image and (ii) interact with the second group of pixels;
determining, by the computing device, that the error or another error did not occur during execution of the automated script; and
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device, the command of the automated script using the adjusted image.
9. The system of claim 8 , wherein modifying the command of the automated script by adjusting the image comprises cropping the image.
10. The system of claim 8 , wherein the operations comprise:
determining that the graphical interface does not include the first group of pixels that match the image,
wherein modifying the command of the automated script is based on determining that the graphical interface does not include the first group of pixels that match the image.
11. The system of claim 8 , wherein determining, by the computing device, that an error occurred during execution of the command of the automated script comprises:
determining that the graphical interface does not include the first group of pixels.
12. The system of claim 8 , wherein the operations comprise:
in response to determining that the error occurred during execution of the command of the automated script, providing, by the computing device, the image and a representation of the graphical interface as an input to a script repair model that is configured to determine an adjustment to the image; and
receiving, from the script repair model, data indicating the adjustment to the image,
wherein adjusting the image is based on the data indicating the adjustment to the image.
13. The system of claim 12 , wherein the operations comprise:
receiving, by the computing device, one or more additional automated scripts that each include commands and results that correspond to each command; and
training, using machine learning, the script repair model using the one or more additional automated scripts that each include commands and the results that correspond to each command.
14. The system of claim 12 , wherein the operations comprise:
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device and using machine learning, the script repair model using the adjusted command and the adjusted image.
15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
providing, for execution and by the computing device, a command of an automated script that is configured to (i) use computer vision to identify, in a graphical interface, a first group of pixels that match an image and (ii) interact with the first group of pixels;
determining, by the computing device, that an error occurred during execution of the command of the automated script;
in response to determining that the error occurred during execution of the command of the automated script, modifying, by the computing device, the command of the automated script by adjusting the image;
providing, for execution and by the computing device, the adjusted command of the automated script that is configured to (i) use computer vision to identify, in the graphical interface, a second group of pixels that match the adjusted image and (ii) interact with the second group of pixels;
determining, by the computing device, that the error or another error did not occur during execution of the automated script; and
based on determining that the error or the other error did not occur during execution of the automated script, updating, by the computing device, the command of the automated script using the adjusted image.
16. The medium of claim 15 , wherein modifying the command of the automated script by adjusting the image comprises cropping the image.
17. The medium of claim 15 , wherein the operations comprise:
determining that the graphical interface does not include the first group of pixels that match the image,
wherein modifying the command of the automated script is based on determining that the graphical interface does not include the first group of pixels that match the image.
18. The medium of claim 15 , wherein determining, by the computing device, that an error occurred during execution of the command of the automated script comprises:
determining that the graphical interface does not include the first group of pixels.
19. The medium of claim 15 , wherein the operations comprise:
in response to determining that the error occurred during execution of the command of the automated script, providing, by the computing device, the image and a representation of the graphical interface as an input to a script repair model that is configured to determine an adjustment to the image; and
receiving, from the script repair model, data indicating the adjustment to the image,
wherein adjusting the image is based on the data indicating the adjustment to the image.
20. The medium of claim 19 , wherein the operations comprise:
receiving, by the computing device, one or more additional automated scripts that each include commands and results that correspond to each command; and
training, using machine learning, the script repair model using the one or more additional automated scripts that each include commands and the results that correspond to each command.