Graphical user interface component identification

1. A non-transitory computer-readable medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:

identifying, by a device, a set of graphical user interface (GUI) components by at least two of:

detecting straight lines in the visual representation of the GUI, where the straight lines create a rectangle enclosing a GUI component;

detecting a geometric shape in the visual representation of the GUI, where the geometric shape encloses a GUI component;

identifying a frequently used shape in the visual representation of the GUI as a GUI component;

matching a region in the visual representation of the GUI to a template describing a commonly used GUI component;

identifying a textual GUI component; and

identifying sub-components of a GUI component;

where the identifying of the set of GUI components is based on screen images that make up a GUI instance and GUI segmentation logic, where the screen images comprise data describing a visual representation of the GUI, and where the set of GUI components comprise one or more GUI elements that facilitate performing a task that the GUI performs;

performing segmentation by using GUI segmentation logic, that is not based on any input from a user, to divide the screen images of the GUI into the set of GUI components;

providing data identifying the set of GUI components in the screen images; and

combining the screen images using the identified set of GUI components into a single screen image of the GUI instance.

2. The non-transitory computer-readable medium of claim 1 , where the data identifying the set of GUI components is provided to one or more of, a data store to store the data identifying the set of GUI components, a content repurposing application, a windows command and control application, a software testing application, a help application, an object recognition application, an application monitoring system, a search engine, and an accessibility application.

3. The non-transitory computer-readable medium of claim 1 , where the data describing the visual representation of the GUI comprises color values describing pixels making up the screen images.

4. The non-transitory computer-readable medium of claim 1 , where a sub-component of a GUI component comprises one or more elements of the GUI component that facilitate performing a task associated with the GUI component, and where identifying a sub-component of a GUI component comprises one or more of:

detecting straight lines in a visual representation of the GUI component, where the straight lines create a rectangle enclosing the sub-component of the GUI component;

identifying a frequently used shape in the visual representation of the GUI component as the sub-component of the GUI component;

matching a region in the visual representation of the GUI component to a template describing a commonly used GUI component;

classifying a textual sub-component of the GUI component; and

recursively identifying sub-components of the sub-component.

5. The non-transitory computer-readable medium of claim 1 , where identifying the set of GUI components is also based on a secondary source of information.

6. The non-transitory computer-readable medium of claim 5 , where the secondary source of information is a document object model (DOM).

7. The non-transitory computer-readable medium of claim 1 , comprising extracting, from the screen images of the GUI, information describing attributes of members of the set of GUI components.

8. A system, comprising:

a processor and memory device including:

image acquisition logic to:

generate graphical user interface (GUI) images of a GUI instant; and

identify a set of operational segments, wherein the set of operational segments is identified by at least two of:

detecting straight lines in the visual representation of the GUI, where the straight lines create a rectangle enclosing a GUI component;

detecting a geometric shape in the visual representation of the GUI, where the geometric shape encloses a GUI component;

identifying a frequently used shape in the visual representation of the GUI as a GUI component;

matching a region in the visual representation of the GUI to a template describing a commonly used GUI component;

identifying a textual GUI component; and

identifying sub-components of a GUI component;

generate a single GUI image by combining the GUI images based on the identified set of operational segments;

GUI segmentation logic that is not based on any user input to divide the GUI images into the set of operational segments, wherein the set of operational segments facilitate performing a task that the GUI instance facilitates; and

provision logic to provide the data identifying the set of operational segments;

where a GUI instant is a state of a GUI between two consecutive inputs to a logic through the GUI.

9. The system of claim 8 , where the GUI images are combined by combining the set of operational segments into the single GUI image.

10. The system of claim 8 , where the GUI segmentation logic comprises one or more of,

layout analysis logic, and

object detection logic.

11. The system of claim 10 , where the layout analysis logic is configured to recursively divide portions of the GUI images of the GUI instance to create a GUI image hierarchy.

12. A computer-implemented method, comprising:

acquiring image data from images describing a graphical user interface (GUI), wherein a GUI instance comprises the images;

automatically identifying a set of GUI components in the image data from the images of the GUI instance by at least two of:

detecting straight lines in the visual representation of the GUI, where the straight lines create a rectangle enclosing a GUI component;

detecting a geometric shape in the visual representation of the GUI, where the geometric shape encloses a GUI component;

identifying a frequently used shape in the visual representation of the GUI as a GUI component;

matching a region in the visual representation of the GUI to a template describing a commonly used GUI component;

identifying a textual GUI component; and

identifying sub-components of a GUI component;

where the automatic identification of the set of GUI components uses segmentation logic and where the set of GUI components comprise one or more GUI elements that facilitate performing a task that the GUI performs;

performing segmentation by using GUI segmentation logic, that is not based on any input from a user, to divide the image data into the set of GUI components;

providing electronic data identifying the set of GUI components;

combining the images into a single image of the GUI instance based on the provided electronic data identifying the set of GUI components.