Framework for UI automation based on graph recognition technology and related methods
A GUI testing device may be configured to execute a testing state machine for interacting with a software application to generate an initial screen of a GUI. The GUI testing device may be configured to determine a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state. The current state may include an operation, and the operation may associate with a trigger target to operate on. The trigger may include a source state, a destination state, and a trigger target. The operation may include a user input operation, and an operation trigger target. The GUI testing device may be configured to perform the operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advance from the current state to a next state based upon the trigger.
1. A computing system comprising:
a graphical user interface (GUI) testing device in communication with a computing device configured to execute a software application with an associated GUI, said GUI testing device comprising a processor and memory cooperating therewith, said processor configured to:
execute a testing state machine for interacting with the software application to generate an initial screen of the GUI, the testing state machine comprising a plurality of states,
determine a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state, the current state comprising at least one operation, and at least one trigger associated with the at least one operation, the at least one trigger including a source state, a destination state, and a trigger target, the at least one operation comprising a user input operation, and an operation trigger target, the determining of the current state comprising:
performing optical character recognition (OCR) on the initial screen to generate a plurality of text strings, and plurality of boxes associated with the plurality of text strings,
finding the trigger including the matching trigger target in the plurality of text strings, the matching trigger target including a source state being the current state, and
merging adjacent text strings in the plurality of text strings, and
perform the at least one operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advance from the current state to a next state based upon the at least one trigger.
2. The computing system of claim 1 wherein said processor is configured to determine a plurality of GUI elements in the initial screen; and wherein the matching trigger target comprises a matching target GUI element from the plurality of GUI elements.
3. The computing system of claim 1 wherein said processor is configured to determine the current state in the testing state machine by at least:
applying a pre-trained convolutional neural network (CNN) to generate a plurality of labels, and a plurality of boxes associated with the plurality of labels; and
finding the trigger including the matching trigger target in the plurality of labels, the matching trigger including a source state being the current state.
4. The computing system of claim 1 wherein said processor is configured to determine the current state in the testing state machine by at least:
determining and applying a matching image template with the initial screen; and
finding the trigger including the matching trigger target in the matching image template, the matching trigger including a source state being the current state.
5. The computing system of claim 1 wherein said processor is configured to perform the at least one operation on the trigger target in a current screen to generate another screen of the GUI.
6. The computing system of claim 1 wherein the at least one operation comprises a plurality thereof; wherein the trigger target comprises a plurality thereof associated with the plurality of operations; and wherein said processor is configured to iteratively perform each operation on the trigger target to generate a plurality of next screens of the GUI.
7. The computing system of claim 1 wherein the user input operation comprises at least one of a keyboard input and a mouse input.
8. The computing system of claim 1 wherein said processor is configured to execute the testing state machine based upon a JavaScript Object Notation (JSON) file.
9. A method for operating a graphical user interface (GUI) testing device in communication with a computing device configured to execute a software application with an associated GUI, the method comprising:
executing a testing state machine for interacting with the software application to generate an initial screen of the GUI, the testing state machine comprising a plurality of states;
determining a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state, the current state comprising at least one operation, and at least one trigger target associated with the at least one operation, the at least one trigger including a source state, a destination state, and a trigger target, the at least one operation comprising a user input operation, and an operation trigger target, the determining comprising:
performing optical character recognition (OCR) on the initial screen to generate a plurality of text strings, and plurality of boxes associated with the plurality of text strings,
finding the trigger including the matching trigger target in the plurality of text strings, the matching trigger including a source state being the current state, and
merging adjacent text strings in the plurality of text strings; and
performing the at least one operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advancing from the current state to a next state based upon the at least one trigger.
10. The method of claim 9 further comprising determining a plurality of GUI elements in the initial screen; and wherein the matching trigger target comprises a matching target GUI element from the plurality of GUI elements.
11. The method of claim 9 wherein determining the current state in the testing state machine comprises:
applying a pre-trained convolutional neural network (CNN) to generate a plurality of labels, and a plurality of boxes associated with the plurality of labels; and
finding the trigger including the matching trigger target in the plurality of labels, the matching trigger including a source state being the current state.
12. The method of claim 9 wherein determining the current state in the testing state machine comprises:
determining and applying a matching image template with the initial screen; and
finding the trigger including the matching trigger target in the matching image template, the matching trigger target including a source state being the current state.
13. The method of claim 9 further comprising performing the at least one operation on the trigger target in a current screen to generate another screen of the GUI.
14. The method of claim 9 wherein the at least one operation comprises a plurality thereof; wherein the trigger target comprises a plurality thereof associated with the plurality of operations; and further comprising iteratively performing each operation on the trigger target to generate a plurality of next screens of the GUI.
15. The method of claim 9 wherein the user input operation comprises at least one of a keyboard input and a mouse input.
16. The method of claim 9 further comprising merging a plurality of text boxes in the initial screen of the GUI.