Self-learning robotic process automation
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a self-learning robotic process automation are disclosed. In one aspect, a method includes receiving an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface. The method further includes executing a command of the one or more commands of the automated script. The method further includes determining that an error occurred during execution of the command of the one or more commands of the automated script. The method further includes determining a modification for the command by applying a script repair model. The method further includes executing the modified command. The method further includes determining whether the error or another error occurred during execution of the modified command. The method further includes determining whether to update the automated script with the modified command.
1. A computer-implemented method comprising:
receiving, by a script executor of a script repair system that includes (i) the script executor, (ii) an error detector, (iii) a command modifier, and (iv) a modified command simulator, an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface on a display of a computer;
executing, by the script executor of the script repair system, a command of the one or more commands of the automated script;
during execution of the command of the one or more commands of the automated script, analyzing, using computer vision and by the script executor of the script repair system, pixels of a screen capture of the user interface on the display of the computer for pixels that match pixels of an image that corresponds to the command;
determining, by the error detector of the script repair system, that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command;
based on determining that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command, determining, by the error detector of the script repair system, that an error occurred during execution of the command of the one or more commands of the automated script;
in response to determining that an error occurred during execution of the command of the one or more commands of the automated script, cropping, by the command modifier of the script repair system, the image that corresponds to the command by removing a portion of the pixels of the image that corresponds to the command to match a group of the pixels of the screen capture of the user interface on the display of the computer;
executing, by the modified command simulator of the script repair system, the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
determining, by the error detector of the script repair system, that the error or another error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
based on determining that the error or the other error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command, updating, by the command modifier of the script repair system, the automated script with the cropped image; and
executing, by the script executor of the script repair system, the automated script with the cropped image.
2. The method of claim 1 , comprising:
based on determining that the error or the other error did not occur during execution of the modified command, updating a script repair model using data indicating that the error or the other error did not occur during execution of the modified command.
3. The method of claim 1 , comprising:
identifying another automated script that includes another image that is similar to the command of the automated script; and
updating the other image of the other automated script with the cropped image.
4. The method of claim 1 , comprising:
classifying, by a script repair model, the error that occurred during execution of the command of the one or more commands of the automated script; and
based on the classification of the error, determining a modification for the command by applying, to the command, the script repair model.
5. The method of claim 4 , comprising:
based on applying the script repair model to the command, determining for each of one or more candidate modifications, a repair confidence score that reflects a likelihood that applying a respective candidate modification to the command will correct the error; and
based on the repair confidence scores, selecting, from among the one or more candidate modifications, the modification for the command.
6. The method of claim 4 , comprising:
receiving one or more automated scripts that each include commands and results that correspond to each command; and
training, using machine learning and the one or more automated scripts that each include commands and results that correspond to each command, the script repair model.
7. The method of claim 1 , wherein:
executing the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command comprises:
simulating selection of the group of pixels of the screen capture, and
determining whether the error or another error occurred during execution of the modified command comprises:
determining whether the error or the other error occurred during simulation of the selection of the group of pixels of the screen capture.
8. A system comprising:
one or more computers; and
one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, by a script executor of a script repair system that includes (i) the script executor, (ii) an error detector, (iii) a command modifier, and (iv) a modified command simulator, an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface on a display of a computer;
executing, by the script executor of the script repair system, a command of the one or more commands of the automated script;
during execution of the command of the one or more commands of the automated script, analyzing, using computer vision and by the script executor of the script repair system, pixels of a screen capture of the user interface on the display of the computer for pixels that match pixels of an image that corresponds to the command;
determining, by the error detector of the script repair system, that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command;
based on determining that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command, determining, by the error detector of the script repair system, that an error occurred during execution of the command of the one or more commands of the automated script;
in response to determining that an error occurred during execution of the command of the one or more commands of the automated script, cropping, by the command modifier of the script repair system, the image that corresponds to the command by removing a portion of the pixels of the image that corresponds to the command to match a group of the pixels of the screen capture of the user interface on the display of the computer;
executing, by the modified command simulator of the script repair system, the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
determining, by the error detector of the script repair system, that the error or another error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
based on determining that the error or the other error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command, updating, by the command modifier of the script repair system, the automated script with the cropped image; and
executing, by the script executor of the script repair system, the automated script with the cropped image.
9. The system of claim 8 , wherein the operations comprise:
based on determining that the error or the other error did not occur during execution of the modified command, updating a script repair model using data indicating that the error or the other error did not occur during execution of the modified command.
10. The system of claim 8 , wherein the operations comprise:
identifying another automated script that includes another image that is similar to the command of the automated script; and
updating the other image of the other automated script with the cropped image.
11. The system of claim 8 , wherein the operations comprise:
classifying, by a script repair model, the error that occurred during execution of the command of the one or more commands of the automated script; and
based on the classification of the error, determining a modification for the command by applying, to the command, the script repair model.
12. The system of claim 11 , wherein the operations comprise:
based on applying the script repair model to the command, determining for each of one or more candidate modifications, a repair confidence score that reflects a likelihood that applying a respective candidate modification to the command will correct the error; and
based on the repair confidence scores, selecting, from among the one or more candidate modifications, the modification for the command.
13. The system of claim 11 , wherein the operations comprise:
receiving one or more automated scripts that each include commands and results that correspond to each command; and
training, using machine learning and the one or more automated scripts that each include commands and results that correspond to each command, the script repair model.
14. The system of claim 8 , wherein:
executing the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command comprises:
simulating selection of the group of pixels of the screen capture, and
determining whether the error or another error occurred during execution of the modified command comprises:
determining whether the error or the other error occurred during simulation of the selection of the group of pixels of the screen capture.
15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
receiving, by a script executor of a script repair system that includes (i) the script executor, (ii) an error detector, (iii) a command modifier, and (iv) a modified command simulator, an automated script that includes one or more commands and that is configured to interact with graphical elements that appear on a user interface on a display of a computer;
executing, by the script executor of the script repair system, a command of the one or more commands of the automated script;
during execution of the command of the one or more commands of the automated script, analyzing, using computer vision and by the script executor of the script repair system, pixels of a screen capture of the user interface on the display of the computer for pixels that match pixels of an image that corresponds to the command;
determining, by the error detector of the script repair system, that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command;
based on determining that the pixels of the screen capture of the user interface on the display of the computer do not include the pixels of the image that corresponds to the command, determining, by the error detector of the script repair system, that an error occurred during execution of the command of the one or more commands of the automated script;
in response to determining that an error occurred during execution of the command of the one or more commands of the automated script, cropping, by the command modifier of the script repair system, the image that corresponds to the command by removing a portion of the pixels of the image that corresponds to the command to match a group of the pixels of the screen capture of the user interface on the display of the computer;
executing, by the modified command simulator of the script repair system, the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
determining, by the error detector of the script repair system, that the error or another error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command;
based on determining that the error or the other error did not occur during selection of the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command, updating, by the command modifier of the script repair system, the automated script with the cropped image; and
executing, by the script executor of the script repair system, the automated script with the cropped image.
16. The medium of claim 15 , wherein the operations comprise:
based on determining that the error or the other error did not occur during execution of the modified command, updating a script repair model using data indicating that the error or the other error did not occur during execution of the modified command.
17. The medium of claim 15 , wherein the operations comprise:
identifying another automated script that includes another image that is similar to the command of the automated script; and
updating the other image of the other automated script with the cropped image.
18. The medium of claim 15 , wherein the operations comprise:
classifying, by a script repair model, the error that occurred during execution of the command of the one or more commands of the automated script; and
based on the classification of the error, determining a modification for the command by applying, to the command, the script repair model.
19. The medium of claim 18 , wherein the operations comprise:
based on applying the script repair model to the command, determining for each of one or more candidate modifications, a repair confidence score that reflects a likelihood that applying a respective candidate modification to the command will correct the error; and
based on the repair confidence scores, selecting, from among the one or more candidate modifications, the modification for the command.
20. The medium of claim 18 , wherein the operations comprise:
receiving one or more automated scripts that each include commands and results that correspond to each command; and
training, using machine learning and the one or more automated scripts that each include commands and results that correspond to each command, the script repair model.
21. The medium of claim 15 , wherein:
executing the command by selecting the group of pixels of the screen capture of the user interface on the display of the computer that match the cropped image that corresponds to the command comprises:
simulating selection of the group of pixels of the screen capture, and
determining whether the error or another error occurred during execution of the modified command comprises:
determining whether the error or the other error occurred during simulation of the selection of the group of pixels of the screen capture.