Test result stability scoring in integration testing
Software code is tested to determine relative performance of the software code. Successfully executing a test may be used to flag problems in a code repository to be corrected or criteria for accepting code for acceptance to the repository. To further evaluate test results, the test may be run several times on the same code version to determine a test result stability score describing how the test results may vary in different executions, such as a frequency that the code passes the test. Based on the score, additional tests may be run to further refine test result stability score. To evaluate time-variance of testing, when a code version fails a test, a previous version of the code may be run with the same test and the results compared to determine if a new regression was introduced.
1 . A method, comprising:
selecting a first number of times to run a test on a version of a software code in a code test environment, wherein the first number of times is inversely related to a first baseline test result stability score;
running the test the first number of times on the version of the software code in the code test environment;
determining first test results of the test being run the first number of times;
determining a first test result stability score based on the first test results, wherein a range of possible test results stability scores may be determined for a confidence value, wherein the confidence value is based on the first number of times the test is run;
in response to determining that the first test result stability score is below the first baseline test result stability score, running the test an additional number of times on the version of the software code in the code test environment until the range of the possible test result stability scores is narrowed to within a given confidence;
determining second test results of the test being run the additional number of times;
determining a second test result stability score based on the first test results and the second test results;
evaluate whether the second test result stability score is greater than a second baseline test result stability score, wherein the second baseline test results stability score is greater than the first baseline test result stability score; and
in response to determining that the second test result stability score is greater than the second baseline test result stability score, cease performing the test on the version of the software code to conserve resources for performing tests.
2 . The method of claim 1 , wherein the confidence of the second test result stability score is based on the first number of times and the additional number of times the test were run on the version of the software code.
3 . The method of claim 1 , wherein the first test result stability score has a range, and running the test an additional number of times narrows the range with the given confidence.
4 . The method of claim 1 , wherein the first test result stability score comprises a pass frequency of the first test results.
5 . The method of claim 1 , wherein the first test result stability score measures a statistical variation of subscores in the first test results, wherein a subscore is used in determining a pass or a fail for a given test.
6 . The method of claim 5 , wherein the subscore is compared against a threshold to determine whether the given test passes or fails.
7 . A system comprising:
a processor; and
a non-transitory computer-readable storage medium comprising instructions for execution by the processor to:
select a first number of times to run a test on a version of a software code, wherein the first number of times is inversely related to a first baseline test result stability score;
run a first type of test the first number of times to test a characteristic of software code;
determine first test results of the first type of test being run the first number of times;
determine a first test result stability score based on the first test results, wherein a range of possible test results stability scores may be determined for a confidence value, wherein the confidence value is based on a number of times the test is run;
in response to determining that the first test result stability score is below a first baseline test result stability score, run a second type of test a second number of times on the software code to test the characteristic until the range of the possible test result stability scores is within a given confidence, wherein the second type of test uses higher resources than the first type of test;
determine second test results of the second type of test being run the second number of times;
determine a second test result stability score based on the second test results; and
evaluate whether the characteristic of the software code passes or fails based on a comparison of the second test result stability score to a second baseline test result stability score, wherein the second baseline test result stability score is greater than the first baseline test result stability score.
8 . The system of claim 7 , wherein the first test result stability score comprises a pass frequency of the first test results.
9 . The system of claim 7 , wherein the second test result stability score comprises a pass frequency of the second test results.
10 . The system of claim 7 , wherein the first test result stability score measures a statistical variation of subscores in the first test results, wherein a subscore is used in determining a pass or a fail for a given test.
11 . The system of claim 7 , wherein the second test result stability score measures a statistical variation of subscores in the second test results, wherein a subscore is used in determining a pass or a fail for a given test.
12 . The system of claim 7 , wherein the instructions cause the processor to further:
accept the software code in response to determining the second test result stability score is above a second baseline test result stability score, wherein the second baseline test result stability score is higher than the first baseline test result stability score.
13 . One or more non-transitory computer-readable storage media comprising instructions executable by one or more processors to:
select a first number of times to run a test on a version of a software code in a code test environment, wherein the first number of times is inversely related to a first baseline test result stability score;
run the test the first number of times on the version of the software code in the code test environment;
determine first test results of the test being run the first number of times;
determine a first test result stability score based on the first test results, wherein a range of possible test results stability scores may be determined for a confidence value, wherein the confidence value is based on a number of times the test is run;
in response to determining that the first test result stability score is below a baseline test result stability score, run the test an additional number of times on the version of the software code in the code test environment until the range of the possible test result stability scores is narrowed to within a given confidence;
determine second test results of the test being run the additional number of times;
determine a second test result stability score based on the first test results and the second test results;
evaluate whether the second test result stability score is greater than a second baseline test result stability score, wherein the second baseline test results stability score is greater than the first baseline test result stability score; and
in response to determining that the second test result stability score is greater than the second baseline test result stability score cease performing the test on the version of the software code to conserve resources for performing tests.
14 . The one or more non-transitory computer-readable storage media of claim 13 , wherein the instructions cause the one or more processors to further:
permit a first code version to be merged to a main code development branch.
15 . The one or more non-transitory computer-readable storage media of claim 13 , wherein the instructions cause the one or more processors to further:
transmit a message to a user to indicate a first code version failed the test but the first code version is accepted due to a previous code version and the first code version having the same performance.
16 . The one or more non-transitory computer-readable storage media of claim 13 , wherein the instructions cause the one or more processors to further:
in response to determining that the first test result stability score and the second test result stability score indicate a previous code version performed better than a first code version, confirm that the first code version has failed the test and run the test an additional number of times on the first code version.
17 . The one or more non-transitory computer-readable storage media of claim 13 , wherein the first test result stability score measures a statistical variation of subscores from running the test on a previous code version multiple times, wherein a subscore is used in determining a pass or a fail for a given test.
18 . The one or more non-transitory computer-readable storage media of claim 13 , wherein the second test result stability score measures a statistical variation of subscores from running the test on a first code version multiple times, wherein a subscore is used in determining a pass or a fail for a given test.