Image based method and system for building object model and application states comparison and graphic-based interoperability with an application
A system and method for enabling graphic-based interoperability between computer executed applications. A computer system operating as a client may display a graphical user interface (GUI) including control graphic items such as buttons, text boxes, etc. A process may examine the graphical image of the GUI to determine if there has been a change over time in the GUI as displayed which updates a control graphic item. If there has been a change over time in the GUI which updates a control graphic item, an action may be taken, for example updating properties of an object construct corresponding to the control graphic item, raising an event corresponding to an object construct corresponding to the control graphic item, or communicating an event to a process.
1. A method for enabling graphic-based interoperability with a computer executed application, the method comprising, on a computer system operating as a client and displaying a graphical user interface (GUI) generated by a third-party application, the GUI comprising control graphic items:
examining changes in a graphical image of the GUI, and based on the examination:
taking an action, the action selected from:
updating properties of an object construct corresponding to a control graphic item; and
raising an event corresponding to an object construct corresponding to a control graphic item.
2. The method as in claim 1 , wherein the event communicates information to the third-party application.
3. The method as in claim 1 , wherein the properties are object-oriented properties.
4. The method as in claim 1 , wherein the event comprises a function which sends a message to a listener or an event handler.
5. The method as in claim 1 , comprising examining the graphical image of the GUI upon an occurrence of the end of a time period, a detection of keyboard input, a detection of mouse input, or an indication that the GUI has changed.
6. The method as in claim 1 , wherein the control graphic item comprises one of a button, a link, a list box, a check box, a table, a drop-list, a combination box, a tab, and a slider.
7. The method as in claim 1 , comprising examining the graphical image of the GUI to identify within the GUI a control graphic item.
8. The method of claim 1 , wherein the third-party application comprises server software modules executed on a computer remote from the computer system.
9. The method of claim 1 , wherein raising an event comprises determining that a state of the control graphic item has changed and notifying a process of the change.
10. A system for enabling graphic-based interoperability with a computer executed application, the system comprising:
a computer system operating as a client comprising a memory and a processor, the processor configured to:
display a graphical user interface (GUI) generated by a third-party application;
examine a graphical image of the GUI to determine if there has been a change in the GUI, and based on the examination take an action, the action selected from:
updating properties of an object construct corresponding to the control graphic item; and
raising an event corresponding to an object construct corresponding to the control graphic item.
11. The system as in claim 10 , wherein the event communicates information to the third-party application.
12. The system as in claim 10 , wherein the properties are object-oriented properties.
13. The system as in claim 10 , wherein the event comprises a function which sends a message to a listener or an event handler.
14. The system as in claim 10 , the processor configured to execute the computer executed application and a second program on the computer system, the second program to perform the examining.
15. The system of claim 10 , wherein raising an event comprises determining that a state of the control graphic item has changed and notifying a process of the change.
16. A method for communication between a first executing software program and a second executing software program, the method comprising:
the first executing software program obtaining a graphical image generated by a third-party application and displayed by the second executing software program;
the first executing software program determining if there has been a change in the obtained graphical image; and
if there has been a change then notifying the third-party application of the change.
17. The method as in claim 16 , comprising determining if there has been a change upon an occurrence of the end of a time period, a detection of keyboard input, a detection of mouse input, or an indication that the GUI has changed.
18. The method as in claim 16 , comprising simulating a functionality of a graphical control in the graphical image based on the type of the graphical control.
19. The method as in claim 16 , comprising examining the obtained graphical image to identify within the obtained graphical image a graphical control.
20. The method of claim 16 , comprising executing the first executing software program and the second executing software program on the same computer.