Granted Patent B2

US 12,332,770 · App. 17/360,390 · Granted Jun 17, 2025

Inventors: Jin Shi (Ningbo, CN); Lu Chen (Ningbo, CN); Tang Xue Bo (Ningbo, CN); Ping Yang (Ningbo, CN); Meng Qi Chen (Ningbo, CN); Rui Na Liu (Ningbo, CN)

Assignee: International Business Machines Corporation

G06F11/3688G06F3/0485G06F11/3698G06N20/00G06T7/70G06T2207/20081

Abstract

Claims (50)

1. A computer-implemented method (CIM) comprising:

receiving a first graphical user interface, including a first plurality of graphical user interface elements;

receiving a machine learning model trained to determine positions of groups of user interface elements and their corresponding constituent user interface elements in an image of a graphical user interface, wherein a serial number assigned to each of the corresponding constituent user interface elements are highlighted on the fly;

indexing, into a first position mapping table, each graphical user interface element of the first plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements, and wherein the multidimensional index value comprises: (i) a first dimension indicative of horizontal position on the first graphical user interface, (ii) a second dimension comprising vertical position on the first graphical user interface, and (iii) a third dimension corresponding to relative placement within the hierarchical group of elements;

testing the first graphical user interface with a first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements for testing based, at least in part, on the serial number as a locator to locate the corresponding constituent user interface elements on the first position mapping table; and

applying a scrolling technique to generate an expanded virtual graphical user interface of one or more groups of user interface elements, wherein the machine learning model refreshes the multidimensional index values using the expanded virtual graphical user interface to reveal other graphical user interface elements not previously visible by: analyzing the expanded virtual graphical user interface (GUI) to identify newly revealed GUI elements; determining positions and hierarchical relationships of the newly revealed GUI elements; and updating the first position mapping table with new multidimensional index values for the newly revealed GUI elements, wherein the new multidimensional index values maintain consistency with an indexing scheme of previously indexed elements while incorporating the newly revealed elements into a hierarchical structure.

2. The CIM of claim 1 , wherein the multidimensional index value further includes a fourth dimension corresponding to a version of the first graphical user interface, where different versions of the first graphical user interface correspond to changes to the first graphical user interface.

3. The CIM of claim 1 , wherein the first automated graphical user interface test further comprises:

executing a first user interface test simulating user interaction between at least some of the first plurality of graphical user interface elements by simulating user input with the first graphical user interface, where the first plurality of graphical user interface elements are located by their respectively assigned multidimensional index value in the first position mapping table.

4. The CIM of claim 1 , further comprising:

receiving a second graphical user interface, based, at least in part, on the first graphical user interface, including a second plurality of graphical user interface elements, where at least some of the second plurality of graphical user interface elements correspond to each of the graphical user interface elements of the first plurality of graphical user interface elements;

indexing, into a second position mapping table, each graphical user interface element of the second plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements; and

testing the second graphical user interface with the first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements based, at least in part, on the second position mapping table.

5. The CIM of claim 4 , further comprising:

while testing the second graphical user interface with the first automated graphical user interface test, responsive to the first graphical user interface test requesting an index absent from the second position mapping table, generating a virtual map of the second graphical user interface by scrolling at least some portions of the second graphical user interface in one or more directions, where the virtual map includes at least some graphical user interface elements of the second plurality of graphical user interface elements which were not visible on the second graphical user interface when previously indexed.

6. A computer program product (CPP) comprising:

a machine readable storage device; and

computer code stored on the machine readable storage device, with the computer code including instructions for causing a processor set to perform operations including the following:

receiving a first graphical user interface, including a first plurality of graphical user interface elements,

receiving a machine learning model trained to determine positions of groups of user interface elements and their corresponding constituent user interface elements in an image of a graphical user interface, wherein a serial number assigned to each of the corresponding constituent user interface elements are highlighted on the fly;

indexing, into a first position mapping table, each graphical user interface element of the first plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements, and wherein the multidimensional index value comprises: (i) a first dimension indicative of horizontal position on the first graphical user interface, (ii) a second dimension comprising vertical position on the first graphical user interface, and (iii) a third dimension corresponding to relative placement within the hierarchical group of elements;

testing the first graphical user interface with a first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements for testing based, at least in part, on the serial number as a locator to locate the corresponding constituent user interface elements on the first position mapping table; and

applying a scrolling technique to generate an expanded virtual graphical user interface of one or more groups of user interface elements, wherein the machine learning model refreshes the multidimensional index values using the expanded virtual graphical user interface to reveal other graphical user interface elements not previously visible by: analyzing the expanded virtual graphical user interface (GUI) to identify newly revealed GUI elements; determining positions and hierarchical relationships of the newly revealed GUI elements; and updating the first position mapping table with new multidimensional index values for the newly revealed GUI elements, wherein the new multidimensional index values maintain consistency with an indexing scheme of previously indexed elements while incorporating the newly revealed elements into a hierarchical structure.

7. The CPP of claim 6 , wherein the multidimensional index value further includes a fourth dimension corresponding to a version of the first graphical user interface, where different versions of the first graphical user interface correspond to changes to the first graphical user interface.

8. The CPP of claim 6 , wherein the first automated graphical user interface test further comprises:

executing a first user interface test simulating user interaction between at least some of the first plurality of graphical user interface elements by simulating user input with the first graphical user interface, where the first plurality of graphical user interface elements are located by their respectively assigned multidimensional index value in the first position mapping table.

9. The CPP of claim 6 , wherein the computer code further includes instructions for causing the processor set to perform the following operations:

receiving a second graphical user interface, based, at least in part, on the first graphical user interface, including a second plurality of graphical user interface elements, where at least some of the second plurality of graphical user interface elements correspond to each of the graphical user interface elements of the first plurality of graphical user interface elements;

indexing, into a second position mapping table, each graphical user interface element of the second plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements; and

testing the second graphical user interface with the first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements based, at least in part, on the second position mapping table.

10. The CPP of claim 9 , wherein the computer code further includes instructions for causing the processor set to perform the following operations:

while testing the second graphical user interface with the first automated graphical user interface test, responsive to the first graphical user interface test requesting an index absent from the second position mapping table, generating a virtual map of the second graphical user interface by scrolling at least some portions of the second graphical user interface in one or more directions, where the virtual map includes at least some graphical user interface elements of the second plurality of graphical user interface elements which were not visible on the second graphical user interface when previously indexed.

11. A computer system (CS) comprising:

a processor set;

a machine readable storage device; and

computer code stored on the machine readable storage device, with the computer code including instructions for causing the processor set to perform operations including the following:

receiving a first graphical user interface, including a first plurality of graphical user interface elements,

receiving a machine learning model trained to determine positions of groups of user interface elements and their corresponding constituent user interface elements in an image of a graphical user interface, wherein a serial number assigned to each of the corresponding constituent user interface elements are highlighted on the fly;

indexing, into a first position mapping table, each graphical user interface element of the first plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements, and wherein the multidimensional index value comprises: (i) a first dimension indicative of horizontal position on the first graphical user interface, (ii) a second dimension comprising vertical position on the first graphical user interface, and (iii) a third dimension corresponding to relative placement within the hierarchical group of elements;

testing the first graphical user interface with a first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements for testing based, at least in part, on the serial number as a locator to locate the corresponding constituent user interface elements on the first position mapping table; and

applying a scrolling technique to generate an expanded virtual graphical user interface of one or more groups of user interface elements, wherein the machine learning model refreshes the multidimensional index values using the expanded virtual graphical user interface to reveal other graphical user interface elements not previously visible by: analyzing the expanded virtual graphical user interface (GUI) to identify newly revealed GUI elements; determining positions and hierarchical relationships of the newly revealed GUI elements; and updating the first position mapping table with new multidimensional index values for the newly revealed GUI elements, wherein the new multidimensional index values maintain consistency with an indexing scheme of previously indexed elements while incorporating the newly revealed elements into a hierarchical structure.

12. The CS of claim 11 , wherein the multidimensional index value further includes a fourth dimension corresponding to a version of the first graphical user interface, where different versions of the first graphical user interface correspond to changes to the first graphical user interface.

13. The CS of claim 11 , wherein the first automated graphical user interface test further comprises:

executing a first user interface test simulating user interaction between at least some of the first plurality of graphical user interface elements by simulating user input with the first graphical user interface, where the first plurality of graphical user interface elements are located by their respectively assigned multidimensional index value in the first position mapping table.

14. The CS of claim 11 , wherein the computer code further includes instructions for causing the processor set to perform the following operations:

receiving a second graphical user interface, based, at least in part, on the first graphical user interface, including a second plurality of graphical user interface elements, where at least some of the second plurality of graphical user interface elements correspond to each of the graphical user interface elements of the first plurality of graphical user interface elements;

indexing, into a second position mapping table, each graphical user interface element of the second plurality of graphical user interface elements with a multidimensional index value corresponding to at least three dimensions based, at least in part, on the machine learning model, where at least one dimension of the at least three dimensions corresponds to grouping status within a hierarchical group of elements; and

testing the second graphical user interface with the first automated graphical user interface test, where the first automated graphical user interface test locates graphical user interface elements based, at least in part, on the second position mapping table.

15. The CS of claim 14 , wherein the computer code further includes instructions for causing the processor set to perform the following operations:

while testing the second graphical user interface with the first automated graphical user interface test, responsive to the first graphical user interface test requesting an index absent from the second position mapping table, generating a virtual map of the second graphical user interface by scrolling at least some portions of the second graphical user interface in one or more directions, where the virtual map includes at least some graphical user interface elements of the second plurality of graphical user interface elements which were not visible on the second graphical user interface when previously indexed.

Assignments (1)

Continuity (1)

References Cited (22)