Automatic identification of types of user interface components

1. A method, comprising:

determining, based on user interface code, a plurality of visible elements of a graphical user interface;

determining, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;

grouping the visible elements into at least first and second groups;

determining types of elements within the first and second groups;

in response to detecting a match between the types of elements within the first and second groups, determining a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes:

determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements; and

determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements; and

storing information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

2. The method of claim 1 , wherein the similarity metric is determined based on respective pixel representations for the first and second groups.

3. The method of claim 2 , wherein the pixel representations are based on the determined bounding boxes for ones of the plurality of visible elements.

4. The method of claim 2 , wherein the pixel representations are based on ones of the plurality of visible elements of the graphical user interface.

5. The method of claim 2 , wherein the pixel representations for the first and second groups are represented using respective matrix data structures; and

further comprising:

flattening the matrix data structures for the first and second groups.

6. The method of claim 5 , further comprising:

transposing the flattened matrix data structures for the first and second groups; and

wherein the determining the similarity metric includes:

determining a first dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the second group;

determining a second dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the first group; and

calculating a result for the similarity metric based on a ratio between the first dot product and the second dot product.

7. The method of claim 6 , wherein the first dot product corresponds to a similarity between the first and second groups of elements, and wherein the second dot product corresponds to a value associated with an exact match.

8. A non-transitory computer-readable medium having instructions stored thereon that are capable of execution by a computing device to perform operations comprising:

determining, based on user interface code, a plurality of visible elements of a graphical user interface;

determining, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;

grouping the visible elements into at least first and second groups;

determining types of elements within the first and second groups;

in response to detecting a match between the types of elements within the first and second groups, determining a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes:

determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements; and

determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements; and

storing information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

9. The non-transitory computer-readable medium of claim 8 , wherein the similarity metric is determined based on respective pixel representations for the first and second groups.

10. The non-transitory computer-readable medium of claim 9 , wherein the pixel representations are based on the determined bounding boxes for ones of the plurality of visible elements.

11. The non-transitory computer-readable medium of claim 9 , wherein the pixel representations are based on ones of the plurality of visible elements of the graphical user interface.

12. The non-transitory computer-readable medium of claim 9 , wherein the pixel representations for the first and second groups are represented using respective matrix data structures;

wherein the operations further comprise:

flattening the matrix data structures for the first and second groups.

13. The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

transposing the flattened matrix data structures for the first and second groups; and

wherein the determining the similarity metric includes:

determining a first dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the second group;

determining a second dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the first group; and

calculating a result for the similarity metric based on a ratio between the first dot product and the second dot product.

14. An apparatus comprising:

one or more processing elements configured to:

determine, based on user interface code, a plurality of visible elements of a graphical user interface;

determine, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;

group the visible elements into at least first and second groups;

determine types of elements within the first and second groups;

in response to detecting a match between the types of elements within the first and second groups, determine a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes:

determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements; and

determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements; and

store information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

15. The apparatus of claim 14 , wherein the similarity metric is determined based on respective pixel representations for the first and second groups.

16. The apparatus of claim 15 , wherein the pixel representations are based on the determined bounding boxes for ones of the plurality of visible elements.

17. The apparatus of claim 15 , wherein the pixel representations are based on ones of the plurality of visible elements of the graphical user interface.

18. The apparatus of claim 15 , wherein the pixel representations for the first and second groups are represented using respective matrix data structures;

wherein the apparatus is further configured to:

flatten the matrix data structures for the first and second groups.

19. The apparatus of claim 18 , wherein the apparatus is further configured to:

transpose the flattened matrix data structures for the first and second groups; and

wherein, to determine the similarity metric, the apparatus is configured to:

determine a first dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the second group;

determine a second dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the first group; and

calculate a result for the similarity metric based on a ratio between the first dot product and the second dot product.