Determining functional and descriptive elements of application images for intelligent screen automation

1. A computer-implemented method for determining functional and descriptive elements of application and interface images, the method comprising:

identifying a first set of elements within one or more images of a user interface, wherein each of the first set of elements is independently selected from the group consisting of: vertical lines, horizontal lines, and rectangular elements;

identifying a second set of elements within the one or more images, wherein each of the second set of elements is independently selected from the group consisting of: radio buttons, icons, and textual elements;

determining one or more logical relationships between the textual elements and non-textual elements of the image;

building a hierarchy comprising some or all of the first set of elements and some or all of the second set of elements in order to form a tree-based data structure representing functionality of the user interface; and

outputting the tree-based data structure to a memory.

2. The computer-implemented method as recited in claim 1 , wherein identifying the first set of elements comprises sequentially searching for different types of elements within the image, wherein the sequential search comprises:

line detection;

check box detection;

rectangular element detection; and

table detection.

3. The computer-implemented method as recited in claim 2 , wherein line detection comprises:

converting the image from a standard color space to a Lab color space;

determining a Euclidean distance between components L, a, and b for a plurality of adjacent pixels within the image;

comparing the Euclidean distance for each pair of adjacent pixels to a minimum distance threshold; and

at least partially in response to determining the Euclidean distance for a given pair of adjacent pixels is greater than the minimum distance threshold, designating the pair of adjacent pixels as a candidate line location.

4. The computer-implemented method as recited in claim 3 , wherein the Euclidean distance is calculated for horizontally adjacent pixels within the image and for vertically adjacent pixels within the image.

5. The computer-implemented method as recited in claim 2 , wherein check box detection comprises:

receiving a set of candidate line elements derived from the one or more images;

searching for intersecting candidate line elements among the set of candidate line elements;

designating as check boxes intersecting lines forming a substantially square shape within a square candidate element of the image characterized by a length from about 10 pixels to about 18 pixels.

6. The computer-implemented method as recited in claim 5 , wherein check box detection further comprises:

filtering from the set of line candidates all lines having a length less than 10 pixels;

filtering from the set of line candidates all lines having a length greater than about 18 pixels;

combining neighboring lines from among the filtered set of line candidates;

searching for intersecting lines among the filtered set of combined line candidates to identify a set of candidate rectangles;

computing an aspect ratio of each candidate rectangle; and

designating as check boxes each set of candidate rectangle having an aspect ratio from about 0.9 to about 1.14.

7. The computer-implemented method as recited in claim 2 , wherein table detection comprises:

computing an area of each of a plurality of candidate rectangles derived from the image;

sorting the plurality of candidate rectangles according to descending area;

determining whether each candidate rectangle comprises multiple candidate rows, each candidate row having a same width as the candidate rectangle and a lesser height than the candidate rectangle;

in response to determining a given candidate rectangle comprises multiple candidate rows:

identify all inner rectangles within the given candidate rectangle;

determine an X-coordinate value for each of the inner rectangles;

compare X-coordinate values for each row of the inner rectangles to X-coordinate values for a topmost row of the inner rectangles; and

defining a table element in the tree-based data structure in response to determining the X-coordinate values for at least one row of the inner rectangles match the X-coordinate values for the topmost row of the inner rectangles.

8. The computer-implemented method as recited in claim 2 , wherein for each step of the sequential search, elements in the image detected as a type of element sought in a prior step of the sequential search are excluded from the search.

9. The computer-implemented method as recited in claim 1 , wherein identifying the second set of elements comprises sequentially searching for different types of elements within the image, wherein the sequential search comprises:

radio button detection;

icon detection; and

textual element detection.

10. The computer-implemented method as recited in claim 9 , wherein radio button detection comprises:

reducing color depth of the image to generate a grayscale rendering of the image;

blurring the grayscale rendering of the image;

detecting a set of candidate edges within the blurred grayscale rendering of the image;

filtering candidate edges having a length greater than a predetermined maximum length threshold from the set of candidate edges;

identifying a set of contours and a corresponding set of bounding boxes within the filtered set of candidate edges;

expanding an area of each bounding box by one pixel in each cardinal direction;

combining overlapping ones of the bounding boxes;

shrinking the area of each bounding box by one pixel in each cardinal direction;

computing an aspect ratio of each bounding box;

removing from the set of bounding boxes any bounding box having an aspect ratio less than 0. about 0.9;

removing from the set of bounding boxes any bounding box having an aspect ratio greater than about 1.1;

removing from the set of bounding boxes any bounding box having a height less than a predetermined minimum height threshold;

removing from the set of bounding boxes any bounding box having a height greater than a predetermined maximum height threshold;

estimating a radius of all remaining bounding boxes in the set of bounding boxes;

counting black pixels within a six-pixel window of a length of the radius of each respective bounding box;

filtering from the set of bounding boxes any bounding box having a black pixel count to radius ratio less than about 5.8; and

designating as candidate radio buttons the remaining bounding boxes in the set of bounding boxes.

11. The computer-implemented method as recited in claim 9 , wherein icon detection comprises:

reducing color depth of the image to generate a grayscale rendering of the image;

blurring the grayscale rendering of the image;

detecting a set of candidate edges within the blurred grayscale rendering of the image;

filtering the set of candidate edges to remove candidate edges having a length greater than a predetermined maximum length threshold;

computing a set of bounding boxes for the remaining candidate edges;

removing from the set of bounding boxes any bounding box corresponding to a previously-identified element;

combining proximate, adjacent bounding boxes within the set of bounding boxes;

removing from the set of bounding boxes any bounding box having a height or a width less than a predetermined minimum threshold;

removing from the set of bounding boxes any bounding box having an aspect ratio greater than a predetermined maximum aspect ratio threshold; and

removing from the set of bounding boxes any bounding box having an aspect ratio less than a predetermined minimum aspect ratio threshold; and

designating all remaining bounding boxes in the set of bounding boxes as candidate icons.

12. The computer-implemented method as recited in claim 9 , wherein each of radio button detection and icon detection comprise contour detection.

13. The computer-implemented method as recited in claim 12 , wherein parameters for contour detection in the context of radio button detection comprise:

applying a Gaussian blur with a kernel size of about 3×3;

a high hysteresis threshold in a range from about 80 to about 100; and

a low hysteresis threshold in a range from about 20 to about 40.

14. The computer-implemented method as recited in claim 13 , wherein radio button detection is performed iteratively;

wherein parameters for contour detection in a second iteration of radio button detection comprise:

applying a standard blur with a kernel size of about 3×3; and

a test ratio of about 5.15; and

wherein parameters for contour detection in a second iteration of radio button detection comprise:

applying no blur; and

a test ratio of about 7.5.

15. The computer-implemented method as recited in claim 12 , wherein parameters for contour detection in the context of icon detection comprise:

not applying any blur;

a high hysteresis threshold in a range from about 40 to about 75; and

a low hysteresis threshold in a range from about 125 to about 175.

16. The computer-implemented method as recited in claim 9 , wherein textual element detection comprises distinguishing the textual elements from the icons using a neural network.

17. The computer-implemented method as recited in claim 16 , wherein distinguishing the textual elements from the icons using the neural network is based at least in part on one or more features selected from the group consisting of: a number of colors feature, a color distance to surrounding area feature, an area of dilated image feature, a minimum color distance to neighboring textual elements feature.

18. The computer-implemented method as recited in claim 1 , wherein determining logical relationships between the textual elements and non-textual elements of the image comprises detecting a plurality of labels corresponding to the plurality of elements.

19. The computer-implemented method as recited in claim 1 , wherein the tree-based data structure identifies: each element in the user interface; an (x, y) coordinate location of one or more vertices of each of the elements in the user interface or a corresponding bounding box; an aspect ratio of each of the elements in the user interface or the corresponding bounding box; a label of some or all of the elements in the user interface; and a logical structure of each of the elements in the user interface.

20. The computer-implemented method as recited in claim 1 , further comprising filtering the first set of elements to remove candidate elements that:

duplicate a root container of a tree-based data structure representing functionality of the user interface;

are adjacent to another candidate element;

adjoin another candidate element;

are an invalid container candidate elements; and/or

conflict with another candidate element.