Image-based automation systems and methods
In one implementation, an image-based automation process includes identifying a graphical object of a graphical user interface and performing an action relative to the graphical object at the graphical user interface. The identifying is based on an image including the graphical object.
1. A non-transitory processor-readable medium storing instructions that when executed by a processor, cause the processor to:
access an instruction describing an action on a graphical user interface displayed on a display;
identify, in a first image of the graphical user interface, a graphical object displayed on the graphical user interface relative to which the action is to be performed, the graphical object having a first appearance in the first image;
perform the action relative to the graphical object at the graphical user interface;
identify, in a second image of the graphical user interface, the graphical object, wherein an appearance of the graphical object is changed in response to performance of the action, the graphical object having a second appearance in the second image; and
generate semantic information related to the first appearance and the second appearance of the graphical object, the semantic information semantically describing an appearance of the graphical object both prior to and after performance of the action such that the action performed relative to the graphical object is identifiable from the semantic information.
2. The non-transitory processor-readable medium of claim 1 , wherein to identify the graphical object displayed on the graphical user interface, the instructions are further to cause the processor to perform a structural analysis of the image to identify structural features of the graphical object and perform semantic analysis of the image to identify text in the graphical object.
3. The non-transitory processor-readable medium of claim 1 , wherein the action is a first action and the instructions are further to cause the processor to perform a second action relative to the graphical object, the first action and the second action associated with a composite action.
4. The non-transitory processor-readable medium of claim 1 , wherein:
the graphical object is an element of a composite graphical object; and
wherein the instructions are further to cause the processor to identify the composite graphical object.
5. The processor-readable medium device of claim 1 , wherein the identifying comprises using changes in the image over time to identify candidate graphical objects that include the graphical object.
6. An image-based automation system comprising circuitry that performs operations, comprising:
a structural analysis module to identify a graphical object within a first image and a second image, wherein the first image includes the graphical object having a first appearance prior to performance of an action relative to the graphical object and a second appearance following performance of the action;
a semantic analysis module to generate semantic information related to the first appearance and the second appearance of the graphical object, the semantic information semantically identifying structural features of the graphical object both prior to and after performance of the action, wherein the action relative to the graphical object is identifiable from the semantic information; and
a temporal analysis module to determine whether another action is associated with the graphical object based on a temporal proximity of the semantic information to the other action.
7. The image-based automation system of claim 6 , wherein
the structural analysis module includes a segmentation engine to identify the graphical object based on structural features of the image.
8. The image-based automation system of claim 6 , wherein
the structural analysis module includes a classification engine to classify the graphical object.
9. The image-based automation system of claim 6 , wherein the action is a first action at a first time and the graphical object is a control of a graphical user interface, further comprising:
an image acquisition module to access the image in response to a second action at a second time after the first time, the second action being an action relative to the graphical object.
10. The image-based automation system of claim 6 , wherein:
the action is a first action at the graphical object at a first time;
the structural analysis module is to identify the graphical object in response to a second action at the graphical object a second time after the first time;
the semantic analysis module is to refine the semantic information after the second time in response to the second action; and
the temporal analysis module is to determine that the first action is associated with the graphical object and to define an action descriptor based on the semantic information.
11. The image-based automation system of claim 6 , the temporal analysis module to determine whether multiple sub-actions should be aggregated into the action with respect to the graphical object.
12. The image-based automation system of claim 6 , wherein the action is a first action, and the temporal analysis module is to aggregate a first semantic description of the first action with a second semantic description of a second action into a third semantic description.
13. The image-based automation system of claim 6 , wherein the temporal analysis module is to determine that a semantic description received in response to the action is a complete action in response to no other semantic descriptions associated with the graphical object being received within a predetermined time period.
14. An image-based automation method, comprising:
identifying, by a processor, a graphical object at an image in response to performance of an action with respect to a graphical user interface on which the graphical object is displayed;
generating, by the processor, semantic information associated with the graphical object, the semantic information semantically identifying structural features of the graphical object both prior to and after performance of the action, wherein the action performed on the graphical object is identifiable from the semantic information; and
storing, by the processor, an action descriptor describing the action and including the semantic information.
15. The image-based automation method of claim 14 , wherein the action is a first action, the method further comprising:
determining that a second action is associated with the graphical object; and
defining the action descriptor based on the first action and the second action.
16. The image-based automation method of claim 14 , wherein
the action descriptor is stored as an instruction at a script file.
17. The image based-automation method of claim 14 ,further comprising:
determining that the action occurred at the graphical object.
18. The image-based automation method of claim 14 , wherein the semantic information includes a semantic description of the graphical object having a first appearance prior to performance of the action and the graphical object having a second appearance following performance of the action, wherein the appearance is at least one of a color, a size, and a shape of the graphical object.
19. The image-based automation method of claim 14 , further comprising determining that the action has occurred based on identifying a change in the image as compared to another image containing the graphical object.
20. The image-based automation method of claim 14 , wherein the identifying comprises analyzing a portion of the image within a predefined distance from the action to identify the graphical object.