Robotic process automation

1. A computer-implemented method comprising:

identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;

obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;

applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;

in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;

for each of the identified activities, generating activity information associated with the activity;

generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and

storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.

2. The method of claim 1 , wherein obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process comprises:

obtaining images taken of the display of the computer from a camera.

3. The method of claim 1 , wherein the one or more images comprise a video.

4. The method of claim 1 , wherein generating activity information associated with the activity comprises:

generating a snapshot of a portion of a screen shown on the display before the activity is performed.

5. The method of claim 1 , wherein generating activity information associated with the activity comprises:

generating a screenshot of a screen shown on the display after the activity is performed.

6. The method of claim 1 , wherein the activities comprise two or more of a key press, a mouse click, a screen touch, a process in a foreground change, or a scene change.

7. The method of claim 1 , comprising:

accessing the process definition; and

automatically instructing the robot to physically interact with the computer or the other computer based on the activities and activity information indicated by the process definition.

8. The method of claim 7 , wherein automatically instructing the robot to interact with the computer or the other computer based on the activities and activity information indicated by the process definition comprises:

identifying a portion of a screen shown on another display of the other computer that visually matches a snapshot indicated by the activity information for a particular activity; and

instructing the robot to physically touch the other display at a location corresponding to the center of the portion.

9. The method of claim 7 , wherein automatically instructing the robot to interact with the other computer based on the activities and activity information indicated by the process definition comprises:

determining that a screen shown on the other display corresponds to a screenshot indicated by the activity information for a particular activity; and

in response to determining that the screen shown on the other display corresponds to a screenshot indicated by the activity information for a particular activity, instructing the robot to physically interact with the other computer based on a subsequent activity indicated by the process definition.

10. The method of claim 7 , wherein automatically instructing the robot to interact with the computer or the other computer based on the activities and activity information indicated by the process definition comprises:

identifying a portion of a screen shown on another display of the other computer that visually matches a snapshot indicated by the activity information for a particular activity; and

instructing the robot to provide an electronic signal to the other computer to receive a click on coordinates corresponding to the center of the portion of the screen that visually matches the snapshot.

11. The method of claim 1 , comprising:

generating a graphical version of the process definition that a person can use to modify the process.

12. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;

obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;

applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;

in response to determining that the change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies the predetermined threshold corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;

for each of the identified activities, generating activity information associated with the activity;

generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and

storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.

13. The system of claim 12 , wherein obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process comprises:

obtaining images taken of the display of the computer from a camera.

14. The system of claim 12 , wherein the one or more images comprise a video.

15. The system of claim 12 , wherein generating activity information associated with the activity comprises:

generating a snapshot of a portion of a screen shown on the display before the activity is performed.

16. The system of claim 12 , wherein generating activity information associated with the activity comprises:

generating a screenshot of a screen shown on the display after the activity is performed.

17. The system of claim 12 , wherein the activities comprise tow or more of a key press, a mouse click, a screen touch, a process in a foreground change, or a scene change.

18. The system of claim 12 , comprising:

accessing the process definition; and

automatically instructing the robot to physically interact with the computer or the other computer based on the activities and activity information indicated by the process definition.

19. The system of claim 18 , wherein automatically instructing the robot to interact with the computer or the other computer based on the activities and activity information indicated by the process definition comprises:

identifying a portion of a screen shown on another display of the other computer that visually matches a snapshot indicated by the activity information for a particular activity; and

instructing the robot to physically touch the other display at a location corresponding to the center of the portion.

20. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

identifying a process that (i) is manually performed by a user interacting with a computer, and (ii) is to be automated for performance by a robot that is configured to interact with the computer or another computer;

obtaining images taken of a display of the computer while the user is interacting with the computer in manually performing the process;

applying a computer vision technique, to the images taken of the display of the computer while the user is interacting with the computer in manually performing the process, to determine that a change in images taken of the display of the computer while the user is interacting with the computer in manually performing the process satisfies a predetermined threshold corresponding to a scene change;

in response to determining that the change in images taken of the display of the computer, while the user is interacting with the computer in manually performing the process satisfies the redetermined threshold cones corresponding to a scene change, identifying a first activity corresponding to receipt of input from the user during the process followed by a second activity corresponding to a scene change during the process;

for each of the identified activities, generating activity information associated with the activity;

generating a process definition for use in causing the robot to automatically perform the process by interacting with the computer or the other computer, where the process definition indicates the first activity corresponding to receipt of the input from the user followed by the second activity corresponding to the scene change and, for each indicated activity, indicates the activity information associated with the activity; and

storing the process definition for later use in causing the robot to automatically perform the process by interacting with the computer or the other computer.