Artificially intelligent self-learning software operating program

1. A method of training a computer program to recognize software application controls comprising:

providing a controller and a memory coupled to the controller, wherein the memory is configured to store program instructions executable by the controller, wherein in response to executing the program instructions, the controller is configured to:

receive a first plurality of images of one or more GUIs of one or more software applications,

analyze the first plurality of images of the one or more GUIs via an image recognition algorithm to identify a position and associate a function for each of one or more user input controls found in the images,

generate a set of execution instructions associated with each of the one or more GUIs, each set of execution instructions including the identified position of the input controls and their associated functions in the associated GUI,

test the accuracy of each set of execution instructions in each GUI by executing the associated function at the identified position for each input control in the execution instructions and analyzing one or more images captured to show the results of the execution by determining whether the execution of the associated function at the identified position for each input control in each set of execution instructions results in the intended function being executed in the GUI,

wherein, when the tested accuracy exceeds a threshold accuracy for the location of controls and associated functions, the execution instruction is identified as live and when the tested accuracy does not exceed a threshold accuracy for the location of controls and associated functions, the controller is configured to receive a second plurality of images of one or more GUIs of one or more software applications to analyze, generate a set of execution instructions, and test the accuracy of each set of execution instructions until the tested accuracy exceeds a threshold accuracy for the location of controls and associated functions.

2. The method of claim 1 , wherein the first plurality of images of the one or more GUIs are further analyzed by a text extraction program to identify a position and associate a function for each of one or more user input controls found in the images.

3. The method of claim 2 , wherein the text extraction program utilizes optical character recognition.

4. The method of claim 1 , wherein the identified position of an input control is a centroid position of the input control.

5. The method of claim 1 , wherein one of the execution instructions clicks a button, inputs text, selects checkboxes, or selects drop down options within an application.

6. The method of claim 1 , wherein an end user performs a keyword search which causes the program instructions to generate an execution instruction; the execution instruction corresponding, at least in part, to one or more terms of the keyword search.

7. The method of claim 1 , wherein the program instructions further include instructions that, when executed, dictate the amount of manual review required before the execution instructions are executed.

8. A computer system to recognize software application controls comprising:

a controller; and

a memory coupled to the controller, wherein the memory is configured to store program instructions executable by the controller;

wherein in response to executing the program instructions, the controller is configured to:

receive a first plurality of images of one or more GUIs of one or more software applications,

analyze the first plurality of images of the one or more GUIs via an image recognition algorithm to identify a position and associate a function for each of one or more user input controls found in the images,

generate a set of execution instructions associated with each of the one or more GUIs, each set of execution instructions including the identified position of the input controls and their associated functions in the associated GUI,

test the accuracy of each set of execution instructions in each GUI by executing the associated function at the identified position for each input control in the execution instructions and analyzing one or more images captured to show the results of the execution by determining whether the execution of the associated function at the identified position for each input control in each set of execution instructions results in the intended function being executed in the GUI,

wherein, when the tested accuracy exceeds a threshold accuracy for the location of controls and associated functions, the execution instruction is identified as live and when the tested accuracy does not exceed a threshold accuracy for the location of controls and associated functions, the controller is configured to receive a second plurality of images of one or more GUIs of one or more software applications to analyze, generate a set of execution instructions, and test the accuracy of each set of execution instructions until the tested accuracy exceeds a threshold accuracy for the location of controls and associated functions.

9. The system of claim 8 , wherein the first plurality of images of the one or more GUIs are further analyzed by a text extraction program to identify a position and associate a function for each of one or more user input controls found in the images.

10. The system of claim 9 , wherein the text extraction program utilizes optical character recognition.

11. The system of claim 8 , wherein the identified position of an input control is a centroid position of the input control.

12. The system of claim 8 , wherein one of the execution instructions clicks a button, inputs text, selects checkboxes, or selects drop down options within an application.

13. The system of claim 8 , wherein an end user performs a keyword search which causes the program instructions to generate an execution instruction; the execution instruction corresponding, at least in part, to one or more terms of the keyword search.

14. The system of claim 8 , wherein the program instructions further include instructions that, when executed, dictate the amount of manual review required before the execution instructions are executed.

15. A method of training a computer program to recognize software application controls comprising:

providing a controller and a memory coupled to the controller, wherein the memory is configured to store program instructions executable by the controller, wherein in response to executing the program instructions, the controller is configured to:

receive a first plurality of images of one or more GUIs of one or more software applications;

analyze the first plurality of images of the one or more GUIs via an image recognition algorithm to identify a position and associate a function for each of one or more user input controls found in the images,

generate a set of execution instructions associated with each of the one or more GUIs, each set of execution instructions including the identified position of the input controls and their associated functions in the associated GUI;

test the accuracy of each set of execution instructions in each GUI by executing the associated function at the identified position for each input control in the execution instructions and analyzing one or more images captured to show the results of the execution by determining whether the execution of the associated function at the identified position for each input control in each set of execution instructions results in the intended function being executed in the GUI,

wherein, when the tested accuracy exceeds a threshold accuracy for the location of controls and associated functions, the execution instruction is identified as live and when the tested accuracy does not exceed a threshold accuracy for the location of controls and associated functions, the controller is configured to receive a second plurality of images of one or more GUIs of one or more software applications to analyze, generate a set of execution instructions, and test the accuracy of each set of execution instructions until the tested accuracy exceeds a threshold accuracy for the location of controls and associated functions; and

execute the live execution instructions in response to receiving one or more voice commands or one or more text commands from a user.

16. The method of claim 15 , wherein the first plurality of images of the one or more GUIs are further analyzed by a text extraction program to identify a position and associate a function for each of one or more user input controls found in the images.

17. The method of claim 15 , wherein the identified position of an input control is a centroid position of the input control.

18. The method of claim 15 , wherein, when executed, one of the execution instructions clicks a button, inputs text, selects checkboxes, or selects drop down options within an application.

19. The method of claim 15 , wherein the one or more voice commands or one or more text commands from a user are received as part of an end user keyword search.

20. The method of claim 15 , wherein the program instructions further include instructions that, when executed, dictate the amount of manual review required before the execution instructions are executed.