Granted Patent B2

US 11,099,972 · App. 16/194,949 · Granted Aug 24, 2021

Inventors: Piotr M. Puszkiewicz (Seattle, WA); Diego Colombo (Farnham, GB)

Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC

G06F11/3664G06F11/3612G06K9/6256G06K9/6268

Abstract

Claims (65)

1. A system for validating a graphical user interface (GUI), the system comprising:

one or more processors; and

one or more memory devices that store program code configured to be executed by the one or more processors, the program code comprising:

a test script launcher configured to:

execute an application comprising the GUI;

execute a test script that interacts with the GUI of the application;

capture a plurality of images representing the GUI of the application at different points in time; and

for each captured image, associate a set of tags that identify expected objects in the captured image; and

a GUI validator configured to:

for each captured image, apply a model that classifies one or more graphical objects in the captured image;

determine, for each captured image, whether each tag of the set of tags associated with the captured image matches one of the one or more classified graphical objects in the captured image;

for each captured image in which each tag of the set of tags associated with the captured image matches one of the one or more classified graphical objects in the captured image, successfully validate the captured image; and

for each captured image in which at least one tag of the set of tags associated with the captured image does not match one of the one or more classified graphical objects in the captured image, cause a validation of the captured image to fail.

2. The system of claim 1 , wherein the GUI validator comprises:

an object detector configured to detect the one or more graphical objects in each captured image; and

an object classifier configured to apply the model to classify each of the one or more graphical objects.

3. The system of claim 1 , wherein the model is trained in a first operating environment in which the GUI of the application has a first representation; and

wherein the application is executed in a second operating environment in which the GUI of the application has a second representation that is different than the first representation.

4. The system of claim 1 , wherein the set of tags for at least one captured image comprises a tag that is based on an executing platform of the application.

5. The system of claim 1 , wherein the test script comprises a plurality of randomized interactions.

6. The system of claim 1 , wherein the model is trained using a supervised learning algorithm that comprises:

bounding a region of at least one image; and

tagging the bounded region with a region identifier.

7. The system of claim 1 , wherein the GUI validator is configured to validate each captured image based on a measure of confidence of a classification of each graphical object identified in the captured image.

8. A method of validating a graphical user interface (GUI), the method comprising:

executing an application comprising the GUI;

executing a test script that interacts with the GUI of the application;

capturing a plurality of images representing the GUI of the application at different points in time;

for each captured image:

associating a set of tags that identify expected objects in the captured image;

applying a model that classifies one or more graphical objects in the captured image; and

determining whether each tag of the set of tags associated with the captured image matches one of the one or more classified graphical objects in the captured image;

for each captured image in which each tag of the set of tags associated with the captured image matches one of the one or more classified graphical objects in the captured image, successfully validating the captured image; and

for each captured image in which at least one tag of the set of tags associated with the captured image does not match one of the one or more classified graphical objects in the captured image, causing a validation of the captured image to fail.

9. The method of claim 8 , wherein the applying a model that classifies one or more graphical objects in each captured image comprises:

detecting the one or more graphical objects in each captured image; and

applying the model to classify each of the one or more graphical objects.

10. The method of claim 8 , wherein the model is trained in a first operating environment in which the GUI of the application has a first representation; and

wherein the application is executed in a second operating environment in which the GUI of the application has a second representation that is different than the first representation.

11. The method of claim 8 , wherein the set of tags for at least one captured image comprises a tag that is based on an executing platform of the application.

12. The method of claim 8 , wherein the test script comprises a plurality of randomized interactions.

13. The method of claim 8 , wherein the model is trained using a supervised learning algorithm that comprises:

bounding a region of at least one image; and

tagging the bounded region with a region identifier.

14. The method of claim 8 , wherein the validating the captured image based on the associated set of tags and the classification of each of the one or more graphical objects in the captured image comprises validating the captured image based on a measure of confidence of a classification of each graphical object identified in the captured image.

15. A computer-readable medium having computer program code recorded thereon that when executed by at least one processor causes the at least one processor to perform a method comprising:

executing an application comprising a graphical user interface (GUI);

executing a test script that interacts with the GUI of the application;

capturing a plurality of images representing the GUI of the application at different points in time;

for each captured image:

associating a set of tags that identify expected objects in the captured image;

applying a model that classifies one or more graphical objects in the captured image; and

determining whether each tag of the set of tags associated with the captured image matches one of the one or more classified graphical objects in the captured image;

for each captured image in which each tag of the set of tags associated with the captured image matches one of the one or more classified graphical objects in the captured image, successfully validating the captured image; and

for each captured image in which at least one tag of the set of tags associated with the captured image does not match one of the one or more classified graphical objects in the captured image, causing a validation of the captured image to fail.

16. The computer-readable medium of claim 15 , wherein the applying a model that classifies one or more graphical objects in each captured image comprises:

detecting the one or more graphical objects in each captured image; and

applying the model to classify each of the one or more graphical objects.

17. The computer-readable medium of claim 15 , wherein the model is trained in a first operating environment in which the GUI of the application has a first representation; and

wherein the application is executed in a second operating environment in which the GUI of the application has a second representation that is different than the first representation.

18. The computer-readable medium of claim 15 , wherein the set of tags for at least one captured image comprises a tag that is based on an executing platform of the application.

19. The computer-readable medium of claim 15 , wherein the test script comprises a plurality of randomized interactions.

20. The computer-readable medium of claim 15 , wherein the model is trained using a supervised learning algorithm that comprises:

bounding a region of at least one image; and

tagging the bounded region with a region identifier.

Assignments (1)

Continuity (1)

Cited By (11)

US 12,190,620 US 12,197,927 US 12,229,547 US 12,259,946 US 12,292,960 US 12,332,772 US 12,423,118 US 12,469,272 US 12,573,227 US 12,602,947 US 12,743,470