Detection of user interface controls via invariance guided sub-control learning

1. A computer-implemented method for detecting one or more user interface objects contained in a screen image of a user interface generated by an application program, the method comprising:

receiving an image file containing one or more user interface objects of a user interface generated by an application program;

detecting sub-control objects in the image file, each sub-control object forming a portion of a user interface object that receives user input;

removing one or more extraneous sub-control objects by detecting and removing sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects; and

correlating sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to form one or more controls that correspond to certain one or more user interface objects, wherein the correlating the sub-control objects includes at least:

imposing an alignment condition to align two sub-control objects that correspond to a user interface object;

imposing a score threshold on each sub-control object and removing each sub-control object having a confidence value that falls below the score threshold;

imposing a distance condition to cause a sub-control object that is closer in distance to a sub-control object in question than another sub-control object to be matched to the sub-control object in question;

combining each set of corresponding matched sub-control objects to generate a corresponding control;

generating a confidence score for each control; and

removing each control that has a confidence score below a threshold to generate a final set of controls that corresponds to user interface objects of the user interface generated by the application program.

2. The computer-implemented method of claim 1 further comprising:

retrieving, for each detected sub-control object, a confidence score associated with the sub-control object; and

wherein removing extraneous sub-control objects further comprises, removing sub-control objects having a confidence score below a first threshold score.

3. The computer-implemented method of claim 1 wherein removing extraneous sub-control objects by detecting and removing sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects comprises, for at least one sub-control object in the identified set of sub-control objects:

identifying sub-control objects that overlap with the sub-control object in the identified set of sub-control objects;

removing sub-control objects that overlap with the sub-control object in the identified set of sub-control objects; and

retrieving a confidence score associated with the sub-control object in the identified set of sub-control objects and removing sub-control objects having a confidence score below a first threshold score.

4. The computer-implemented method of claim 1 wherein correlating sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to generate controls that correspond to certain of the user interface objects of the user interface generated by the application program, comprises:

accessing a correlation map that provides a mapping between user interface objects of the user interface and sub-control objects associated with each user interface object.

5. The computer-implemented method of claim 1 wherein generating a confidence score for each control comprises generating the confidence score for each control as a function of confidence scores associated with each sub-control object that forms the control.

6. The computer-implemented method of claim 1 wherein the image file contains an image representation of the one or more user interface objects of the user interface generated by the application program; and

wherein detecting sub-control objects in the image file comprises detecting the sub-control objects from the image representation of the sub-control objects.

7. A computer system comprising:

data storage having stored thereupon a plurality of image files, each image file containing one or more user interface objects of a user interface generated by an application program; and

a processor, programmed with instructions that cause the processor to detect one or more user interface objects contained in a screen image of a user interface generated by an application program by way of invariance guided sub-control learning by:

receiving an image file containing one or more user interface objects of a user interface generated by the application program;

detecting sub-control objects in the image file, each sub-control object forming a portion of a user interface object that receives user input;

removing sub-control objects by detecting and removing sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects; and

correlating sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to generate controls that correspond to certain of the user interface objects of the user interface generated by the application program, wherein the correlating the sub-control objects includes at least:

imposing an alignment condition to align two sub-control objects that correspond to a user interface object;

imposing a score threshold on each sub-control object and removing each sub-control object having a confidence value that falls below the score threshold;

imposing a distance condition to cause a sub-control object that is closer in distance to a sub-control object in question than another sub-control object to be matched to the sub-control object in question;

combining each set of corresponding matched sub-control objects to generate a corresponding control;

generating a confidence score for each control; and

removing each control that has a confidence score below a threshold to generate a final set of controls that corresponds to user interface objects of the user interface generated by the application program.

8. The computer system of claim 7 wherein the processor is further programmed with instructions that cause the processor to detect one or more user interface objects contained in a screen image of a user interface generated by:

retrieving, for each detected sub-control object, a confidence score associated with the sub-control object; and

wherein removing extraneous sub-control objects further comprises, removing sub-control objects having a confidence score below a first threshold score.

9. The computer system of claim 7 wherein removing extraneous sub-control objects by detecting and removing sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects comprises, for at least one sub-control object in the identified set of sub-control objects is performed by:

identifying sub-control objects that overlap with the sub-control object in the identified set of sub-control objects;

removing sub-control objects that overlap with the sub-control object in the identified set of sub-control objects; and

retrieving a confidence score associated with the sub-control object in the identified set of sub-control objects and removing sub-control objects having a confidence score below a first threshold score.

10. The computer system of claim 7 wherein correlating sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to generate controls that correspond to certain of the user interface objects of the user interface generated by the application program is performed by:

accessing a correlation map that provides a mapping between user interface objects of the user interface and sub-control objects associated with each user interface object.

11. The computer system of claim 7 wherein generating a confidence score for each control comprises generating the confidence score for each control as a function of confidence scores associated with each sub-control object that forms the control.

12. A tangible storage medium, having stored thereupon one or more program modules comprising computer-executable instructions for execution on a computer system, the computer-executable instructions executing on a server processor to cause the computer system to perform a computer-implemented method for detecting one or more user interface objects contained in a screen image of a user interface generated by an application program, the computer-implemented method comprising:

receiving an image file containing one or more user interface objects of a user interface generated by an application program;

detecting sub-control objects in the image file, each sub-control object forming a portion of a user interface object that receives user input;

removing extraneous sub-control objects by detecting and removing sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects; and

correlating sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to generate controls that correspond to certain of the user interface objects of the user interface generated by the application program, wherein the correlating the sub-control objects includes at least:

imposing an alignment condition to align two sub-control objects that correspond to a user interface object;

imposing a score threshold on each sub-control object and removing each sub-control object having a confidence value that falls below the score threshold;

imposing a distance condition to cause a sub-control object that is closer in distance to a sub-control object in question than another sub-control object to be matched to the sub-control object in question;

combining each set of corresponding matched sub-control objects to generate a corresponding control;

generating a confidence score for each control; and

removing each control that has a confidence score below a threshold to generate a final set of controls that corresponds to user interface objects of the user interface generated by the application program.

13. The tangible storage medium of claim 12 wherein the computer-implemented method further comprises:

retrieving, for each detected sub-control object, a confidence score associated with the sub-control object; and

wherein removing extraneous sub-control objects further comprises, removing sub-control objects having a confidence score below a first threshold score.

14. The tangible storage medium of claim 12 wherein removing extraneous sub-control objects by detecting and removing sub-control objects that overlap with or that are within a predetermined vicinity of an identified set of sub-control objects comprises, for at least one sub-control object in the identified set of sub-control objects:

identifying sub-control objects that overlap with the sub-control object in the identified set of sub-control objects;

removing sub-control objects that overlap with the sub-control object in the identified set of sub-control objects; and

retrieving a confidence score associated with the sub-control object in the identified set of sub-control objects and removing sub-control objects having a confidence score below a first threshold score.

15. The tangible storage medium of claim 12 wherein correlating sub-control objects in the identified set of sub-control objects to combine one or more of the sub-control objects in the identified set of sub-control objects to generate controls that correspond to certain of the user interface objects of the user interface generated by the application program, comprises:

accessing a correlation map that provides a mapping between user interface objects of the user interface and sub-control objects associated with each user interface object.