IP Library Granted Patent US 12681843
Granted Patent B1
US 12681843 · App. 18/600,485 · Granted Jul 14, 2026

Generating software testing recommendations using machine learning

Inventors: William Christopher Watkins (Seattle, WA); Sean Chickosky (Clinton, MA); Christopher Murray Hind (Kent, WA)
Assignee: Amazon Technologies, Inc.
G06F11/3688G06F8/71
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12681843
App. No.
18/600,485
Granted
Jul 14, 2026
Kind
B1
Abstract

Testing recommendations for an updated software build can be generated. For example, a computer system can determine a classification of a modification of a component of a software build into a category based on a text file that indicates the modification of the component. The computer system can also determine a recognition of an entity type for the component of the software build based on the text file. The computer system can generate, based at least in part on the classification and the recognition, a recommendation indicating one or more tests for use in testing the modification to the software build.

Claims (66)

1 . A computer system, comprising:

a processing device; and

a non-transitory memory that stores instructions that are executable by the processing device for causing the processing device to at least:

receive a text file indicating one or more modifications to a video game build;

provide the text file as a first input to a first machine learning model configured to generate, based at least in part on the first input, a first output comprising a classification of the one or more modifications to the video game build into a category;

receive, from the first machine learning model, the first output;

provide the text file as a second input to a second machine learning model configured to, based at least in part on the second input, identify a component of the video game build in the text file and generate a second output comprising a recognition of an entity type for the component;

receive, from the second machine learning model, the second output;

identify one or more tests from a testing database, wherein the one or more tests are associated with the classification of the one or more modifications or the recognition of the entity type;

output a recommendation indicating the one or more tests to use in testing the one or more modifications to the video game build; and

automatically execute, based at least in part on the recommendation, the one or more tests for testing the one or more modifications to the video game build.

2 . The computer system of claim 1 , wherein the non-transitory memory further stores instructions that are executable by the processing device for causing the processing device to at least:

generate a result for the one or more tests based at least in part on automatically executing the one or more tests.

3 . The computer system of claim 1 , wherein the classification comprises one of a plurality of predefined categories including a user interface category, an art style category, an audio category, an engine category, a game service category, or a platform category for the video game build.

4 . The computer system of claim 1 , wherein the recognition comprises one of a plurality of predefined entity types including a card entity, a player character entity, a non-player character entity, a gear entity, a quest entity, an event entity, or a location entity in the video game build.

5 . A computer-implemented method, comprising:

determining a classification of a modification of a component of a software build into a category based on a text file that indicates the modification of the component;

determining a recognition of an entity type for the component of the software build based on the text file;

generating, based at least in part on the classification and the recognition, a recommendation indicating one or more tests for use in testing the modification to the software build; and

automatically executing, based at least in part on the recommendation, the one or more tests for testing the modification to the software build.

6 . The computer-implemented method of claim 5 , wherein determining the classification and determining the recognition further comprises:

providing the text file as a first input to a first machine learning model configured to generate, based at least in part on the first input, a first output comprising the classification of the modification of the component;

providing the text file as a second input to a second machine learning model configured to generate, based at least in part on the second input, a second output comprising the recognition of the entity type for the component of the software build; and

receiving the first output from the first machine learning model and the second output from the second machine learning model.

7 . The computer-implemented method of claim 6 , wherein the second machine learning model is further configured to generate the second output by:

identifying the component referenced in the text file; and

selecting, from a list of predefined entity types, the entity type that is associated with the component.

8 . The computer-implemented method of claim 6 , wherein the first machine learning model is further configured to generate a first confidence score for the classification, and wherein the second machine learning model is further configured to generate a second confidence score for the recognition.

9 . The computer-implemented method of claim 8 , further comprising:

determining that the first confidence score and the second confidence score each exceed a predefined threshold, and

wherein automatically executing the one or more tests is responsive to determining that the first confidence score and the second confidence score each exceed the predefined threshold.

10 . The computer-implemented method of claim 6 , wherein the first output generated by the first machine learning model further comprises a first predicted number of modifications in the classification for the software build, and wherein the second output generated by the second machine learning model further comprises a second predicted number of modifications for the entity type in the software build.

11 . The computer-implemented method of claim 5 , wherein generating the recommendation indicating the one or more tests further comprises:

providing at least one of the entity type, the classification, or the text file as a third input to a third machine learning model configured to generate, based at least in part on the third input, a third output comprising a ranking of tests from a testing database according to a relevancy score determined between the entity type and each test in the ranking of tests; and

selecting the one or more tests for the recommendation based on the one or more tests having a relevancy score that exceeds a predefined threshold.

12 . The computer-implemented method of claim 11 , further comprising:

determining that none of the tests in the testing database have a relevancy score that exceeds the predefined threshold; and

generating another recommendation to generate a test associated with the classification or the text file in response to determining that none of the tests have the relevancy score that exceeds the predefined threshold.

13 . The computer-implemented method of claim 5 , further comprising:

determining, based at least in part on the classification and the recognition, a queue of tests from a testing database for the recommendation;

detecting that a first test from the queue of tests has been executed; and

removing the first test from the queue of tests.

14 . The computer-implemented method of claim 13 , further comprising:

determining that a second test from the queue of tests has generated a fail result; and

selecting a third test to add to the queue of tests in response to determining that the second test has generated the fail result.

15 . The computer-implemented method of claim 5 , further comprising:

identifying a test suite comprising a plurality of tests in a testing database, the test suite including the one or more tests;

determining that the one or more tests exceed a threshold percentage of the plurality of tests in the test suite; and

generating the recommendation indicating each test of the plurality of tests in the test suite responsive to determining that the one or more tests exceed the threshold percentage.

16 . The computer-implemented method of claim 15 , further comprising:

determining that the one or more tests do not exceed the threshold percentage of the plurality of tests in the test suite; and

generating the recommendation indicating the one or more tests responsive to determining that the one or more tests do not exceed the threshold percentage, wherein the recommendation does not comprise each test of the plurality of tests in the test suite.

17 . One or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by one or more processing devices of a computer system, cause the computer system to perform operations comprising:

determining a classification of one or more modifications of a component of a software build into a category based on a text file that indicates the one or more modifications of the component;

determining a recognition of an entity type for the component of the software build based at least in part on the text file;

generating, based at least in part on the classification and the recognition, a recommendation indicating one or more tests for use in testing the one or more modifications to the software build; and

automatically executing, based at least in part on the recommendation, the one or more tests for use in testing the one or more modifications to the software build.

18 . The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:

generating a result for the one or more tests based at least in part on automatically executing the one or more tests.

19 . The one or more non-transitory computer-readable media of claim 17 , wherein the computer-executable instructions, when executed by the one or more processing devices of the computer system, further cause the computer system to perform the operations comprising:

identifying the component referenced in the text file;

generating a confidence score for an association between the component and each entity type of a list of predefined entity types; and

selecting the entity type from the list of predefined entity types by determining a highest confidence score.

20 . The one or more non-transitory computer-readable media of claim 17 , wherein a first machine learning model is configured to generate the classification and a second machine learning model is configured to generate the recognition, and wherein the operations further comprise:

receiving the text file via a graphical user interface; and

displaying, via the graphical user interface, the classification, the recognition of the entity type, the recommendation indicating the one or more tests, a first confidence score generated by the first machine learning model for the classification, and a second confidence score generated by the second machine learning model for the recognition of the entity type.