Surface automation in black box environments

1. A computer-implemented method, comprising:

determining a scenario of an application for automation, the scenario comprising a user-executable process of the application;

detecting the scenario during an execution of the application;

capturing and storing one or more user interface screenshots of the application taken during execution of the scenario;

identifying and storing user interface information from the one or more user interface screenshots comprising at least target textual elements and target non-textual elements of the one or more user interface screenshots, wherein the user interface information comprises at least one of coordinates of the target textual elements or coordinates of the target non-textual elements;

demarcating the target non-textual elements of the one or more user interface screenshots, wherein the demarcated target non-textual elements of the one or more user interface screenshots are a plurality of templates;

storing the plurality of templates;

implementing a sequential set of instructions to automate the scenario, the sequential set of instructions comprising:

at least one textual element detection technique, wherein a pre-processing image technique identifies textual elements that match target textual elements from the one or more user interface screenshots; and

at least one non-textual element detection technique, wherein the at least one non-textual element detection technique includes template matching comprising

sliding a template from the plurality of templates through a plurality of positions of an automated runtime image,

computing a similarity score of the template at each position, wherein the similarity score determines how similar each position in the plurality of positions in the automated runtime image is to the template, and

selecting a highest score from the similarity scores to determine a most likely position of a non-textual element of the automated runtime input image that matches the target non-textual element associated with the template; and

executing the sequential set of instructions, wherein the execution of the sequential set of instructions comprises automatically implementing the scenario of the application.

2. The method of claim 1 , wherein the at least one textual element detection technique comprises:

using a first pre-processing image technique that identifies textual elements from an automated runtime input image; and

using a second pre-processing image technique that is different from the first pre-processing image technique that identifies textual elements within the automated runtime input image that the first pre-processing image technique may not have identified.

3. The method of claim 2 , wherein the at least one textual detection technique further comprises:

iterating through an object model specific to one of the one or more user interface screenshots in a sequential manner to detect a textual element from the automated runtime input image that matches a target textual element from the one or more user interface screenshots after using the second pre-processing technique.

4. The method of claim 1 , wherein the at least one non-textual element detection technique further comprises:

locating a textual element in an automation runtime input image that matches a textual element stored in association with a template in the plurality of templates;

utilizing a first distance associated with the stored textual element to locate a first reference point in the automation runtime image;

utilizing a second distance associated with the stored textual element to locate a second reference point in the automation runtime image;

matching the template to a first non-textual element of the automation runtime image based on the first reference point; and

matching the template to a second non-textual element of the automation runtime image based on the second reference point.

5. The method of claim 1 , wherein the at least one textual element detection technique comprises:

detecting a plurality of textual elements in an automated runtime input image that match a target textual element from the one or more user interface screenshots;

identifying one of the plurality of textual elements in the automated runtime image that is nearest to stored coordinates associated with the target textual element within a rectangle defined based on the stored coordinates associated with the target textual element, a height parameter, and a width parameter, wherein the height parameter and the width parameter collectively specify a percent deviation of a control location between a recording phase and a recognition phase.

6. A system, comprising:

a memory; and

at least one processor coupled to the memory and configured to:

determine a scenario of an application for automation, the scenario comprising a user-executable process of the application;

detect the scenario during an execution of the application;

capture and store one or more user interface screenshots of the application taken during execution of the scenario;

identify and store user interface information from the one or more user interface screenshots comprising at least target textual elements and target non-textual elements of the one or more user interface screenshots, wherein the user interface information comprises at least one of coordinates of the target textual elements or coordinates of the target non-textual elements;

demarcate the target non-textual elements of the one or more user interface screenshots, wherein the demarcated target non-textual elements of the one or more user interface screenshots are a plurality of templates;

store the plurality of templates;

implement a sequential set of instructions to automate the scenario, the sequential set of instructions comprising:

a textual element detection technique, wherein a pre-processing image technique identifies textual elements that match target textual elements from the one or more user interface screenshots; and

a non-textual element detection technique, wherein the non-textual element detection technique includes template matching comprising

sliding a template from the plurality of templates through a plurality of positions of an automated runtime image,

computing a similarity score of the template at each position, wherein the similarity score determines how similar each position in the plurality of positions in the automated runtime image is to the template, and

selecting a highest score from the similarity scores to determine a most likely position of a non-textual element of the automated runtime input image that matches the target non-textual element associated with the template; and

execute the sequential set of instructions, wherein the execution of the sequential set of instructions comprises automatically implementing the scenario of the application.

7. The system of claim 6 , wherein the textual element detection technique comprises:

using a first pre-processing image technique that identifies textual elements from an automated runtime input image; and

using a second pre-processing image technique that is different from the first pre-processing image technique that identifies textual elements within the automated runtime input image that the first pre-processing image technique may not have identified.

8. The system of claim 7 , wherein the textual element detection technique further comprises:

iterating through an object model specific to one of the one or more user interface screenshots in a sequential manner to detect a textual element from the automated runtime input image that matches a target textual element from the one or more user interface screenshots after using the second pre-processing technique.

9. The system of claim 6 , wherein the non-textual element detection technique further comprises:

locating a textual element in an automation runtime input image that matches a textual element stored in association with a template in the plurality of templates;

utilizing a first distance associated with the stored textual element to locate a first reference point in the automation runtime image;

utilizing a second distance associated with the stored textual element to locate a second reference point in the automation runtime image;

matching the template to a first non-textual element of the automation runtime image based on the first reference point; and

matching the template to a second non-textual element of the automation runtime image based on the second reference point.

10. The system of claim 6 , wherein the textual element detection technique comprises:

detecting a plurality of textual elements in an automated runtime input image that match a target textual element from the one or more user interface screenshots;

identifying one of the plurality of textual elements in the automated runtime image that is nearest to stored coordinates associated with the target textual element within a rectangle defined based on the stored coordinates associated with the target textual element, a height parameter, and a width parameter, wherein the height parameter and the width parameter collectively specify a percent deviation of a control location between a recording phase and a recognition phase.

11. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:

determining a scenario of an application for automation, the scenario comprising a user-executable process of the application;

detecting the scenario during an execution of the application;

capturing and storing one or more user interface screenshots of the application taken during execution of the scenario;

identifying user interface information from the one or more user interface screenshots comprising at least target textual elements and target non-textual elements of the one or more user interface screenshots, wherein the user interface information comprises at least one of coordinates of the target textual elements or coordinates of the target non-textual elements;

demarcating the target non-textual elements of the one or more user interface screenshots, wherein the demarcated target non-textual elements of the one or more user interface screenshots are a plurality of templates;

storing the plurality of templates;

implementing a sequential set of instructions to automate the scenario, the sequential set of instructions comprising:

a textual element detection technique, wherein a pre-processing image technique identifies textual elements that match target textual elements from the one or more user interface screenshots; and

a non-textual element detection technique, wherein the non-textual element detection technique includes template matching comprising

sliding a template from the plurality of templates through a plurality of positions of an automated runtime image,

computing a similarity score of the template at each position, wherein the similarity score determines how similar each position in the plurality of positions in the automated runtime image is to the template, and

selecting a highest score from the similarity scores to determine a most likely position of a non-textual element of the automated runtime input image that matches the target non-textual element associated with the template; and

executing the sequential set of instructions, wherein the execution of the sequential set of instructions comprises automatically implementing the scenario of the application.

12. The non-transitory computer-readable medium of claim 11 , wherein the textual element detection technique comprises:

using a first pre-processing image technique that identifies textual elements from an automated runtime input image; and

using a second pre-processing image technique that is different from the first pre-processing image technique that identifies textual elements within the automated runtime image that the first pre-processing image technique may not have identified.

13. The non-transitory computer-readable medium of claim 12 , wherein the textual element detection technique comprises:

iterating through an object model specific to one of the one or more user interface screenshots in a sequential manner to detect a textual element from the automated runtime input image that matches a target textual element from the one or more user interface screenshots after using the second pre-processing technique.

14. The non-transitory computer-readable medium of claim 11 , wherein the non-textual element detection technique further comprises: locating a textual element in an automation runtime input image that matches a textual element stored in association with a template in the plurality of templates; utilizing a first distance associated with the stored textual element to locate a first reference point in the automation runtime image; utilizing a second distance associated with the stored textual element to locate a second reference point in the automation runtime image; matching the template to a first non-textual element of the automation runtime image based on the first reference point; and matching the template to a second non-textual element of the automation runtime image based on the second reference point.

15. The non-transitory computer-readable medium of claim 11 , wherein the textual element detection technique comprises:

detecting a plurality of textual elements in an automated runtime input image that match a target textual element from the one or more user interface screenshots;

identifying one of the plurality of textual elements in the automated runtime image that is nearest to stored coordinates associated with the target textual element within a rectangle defined based on the stored coordinates associated with the target textual element, a height parameter, and a width parameter, wherein the height parameter and the width parameter collectively specify a percent deviation of a control location between a recording phase and a recognition phase.

16. The method of claim 2 , wherein the first pre-processing image technique comprises a stroke width transform technique and the second pre-processing image technique comprises a binary inversion technique.

17. The system of claim 7 , wherein the first pre-processing image technique comprises a stroke width transform technique and the second pre-processing image technique comprises a binary inversion technique.

18. The method of claim 1 , wherein sliding the template from the plurality of templates through the plurality of positions of the automated runtime image comprises sliding the template from the plurality of templates across the entire automated runtime image.

19. The system of claim 6 , wherein sliding the template from the plurality of templates through the plurality of positions of the automated runtime image comprises sliding the template from the plurality of templates across the entire automated runtime image.

20. The non-transitory computer-readable medium of claim 11 , wherein sliding the template from the plurality of templates through the plurality of positions of the automated runtime image comprises sliding the template from the plurality of templates across the entire automated runtime image.