Dynamic testing based on automated impact analysis

1. A method for automated software testing, comprising:

instrumenting a codebase to extract a plurality of lines in a plurality files that are executed when executing the codebase;

in response to executing the codebase, mapping a plurality of test cases to the plurality of lines,

wherein each test case of the plurality of test cases covers one or more lines in the plurality of lines and has a corresponding plurality of test results;

obtaining a change list comprising one or more changes, each change specifying a changed line in a changed file of the plurality of files;

determining, based on the mapping and the change list, a plurality of impacted test cases of the plurality of test cases by:

attempting to identify, for each change of the change list, one of the plurality of test cases that covers the changed line specified by the change;

failing to identify a test case that covers the changed line specified by a first change of the change list; and

identifying one of the plurality of test cases that covers an unchanged line in the changed file specified by the first change and fails to cover the changed line;

prioritizing the plurality of impacted test cases to favor frequently failing test cases based, in part, on a plurality of test results corresponding to each impacted test case of the plurality of impacted test cases,

wherein the plurality of test results comprises a failure count and a measure of the utilization of a resource of a computer system during execution of the corresponding impacted test case, and

wherein the prioritization is further based on a first pre-determined weighting factor of the measure of the utilization and a second pre-determined weighting factor of the failure count;

executing, based on the prioritization, one or more impacted test cases of the plurality of impacted test cases to obtain a plurality of executed test cases; and

obtaining, for each executed test case of the plurality of executed test cases, a new plurality of test results.

2. The method of claim 1 , further comprising:

storing, in a database, the mapping and a plurality of test results corresponding to each test case of the plurality of test cases; and

updating, for each executed test case, the database with the new plurality of test results.

3. The method of claim 1 ,

wherein each test result of the plurality of test results comprises an execution time, and

wherein the prioritization is further based on a third pre-determined weighting factor of the execution time.

4. The method of claim 1 , wherein the determination is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

5. The method of claim 1 , wherein the prioritization is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

6. The method of claim 1 , further comprising:

issuing an alert identifying one or more lines in one or more files of the plurality of files based on one or more test results corresponding to a test case of the plurality of test cases.

7. A system for automated software testing, comprising:

mapper circuitry configured for: in response to executing a codebase, mapping a plurality of test cases to a plurality of lines in a plurality files in the codebase, wherein each test case of the plurality of test cases covers one or more lines in the plurality of lines and has a corresponding plurality of test results;

change list analyzer circuitry configured for:

obtaining a change list comprising one or more changes, each change specifying a changed line in a changed file of the plurality of files, and

determining, based on the mapping and the change list, a plurality of impacted test cases of the plurality of test cases by:

attempting to identify, for each change of the change list, one of the plurality of test cases that covers the changed line specified by the change

failing to identify a test case that covers the changed line specified by a first change of the change list; and

identifying one of the plurality of test cases that covers an unchanged line in the changed file specified by the first change and fails to cover the changed line;

test sequencer circuitry configured for:

prioritizing the plurality of impacted test cases to favor frequently failing test cases based, in part, on a plurality of test results corresponding to each impacted test case of the plurality of impacted test cases,

wherein the plurality of test results comprises a failure count and a measure of the utilization of a resource of a computer system during execution of the corresponding impacted test case, and

wherein the prioritization is further based on a first pre-determined weighting factor of the measure of the utilization and a second pre-determined weighting factor of the failure count;

test executive circuitry configured for:

instrumenting the codebase to extract the plurality of lines in the plurality files that are executed when executing the codebase;

executing, based on the prioritization, one or more impacted test cases of the plurality of impacted test cases to obtain a plurality of executed test cases, and

obtaining, for each executed test case of the plurality of executed test cases, a new plurality of test results; and

repository circuitry configured for storing at least the codebase and the plurality of test cases.

8. The system of claim 7 ,

wherein the repository circuitry further comprises a database storing the mapping and a plurality of test results corresponding to each test case of the plurality of test cases, and

wherein the test executive circuitry further comprises updating, for each executed test case, the database with each new plurality of test results.

9. The system of claim 7 ,

wherein each test result of the plurality of test results comprises a failure count and an execution time, and

wherein the prioritization is further based on a third pre-determined weighting factor of the execution time.

10. The system of claim 7 , wherein the determination is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

11. The system of claim 7 , wherein the prioritization is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

12. The system of claim 7 , further comprising:

failure analyzer circuitry configured for issuing an alert identifying one or more lines in one or more files of the plurality of files based on one or more test results corresponding to a test case of the plurality of test cases.

13. A non-transitory computer readable medium comprising instructions that, when executed by a hardware processor, perform a method for automated software testing, the method comprising:

instrumenting a codebase to extract a plurality of lines in a plurality files that are executed when executing the codebase;

in response to executing the codebase, mapping a plurality of test cases to the plurality of lines,

wherein each test case of the plurality of test cases covers one or more lines in the plurality of lines and has a corresponding plurality of test results;

obtaining a change list comprising one or more changes, each change specifying a changed line in a changed file of the plurality of files;

determining, based on the mapping and the change list, a plurality of impacted test cases of the plurality of test cases by:

attempting to identify, for each change of the change list, one of the plurality of test cases that covers the changed line specified by the change;

failing to identify a test case that covers the changed line specified by a first change of the change list; and

identifying one of the plurality of test cases that covers an unchanged line in the changed file specified by the first change and fails to cover the changed line;

prioritizing the plurality of impacted test cases to favor frequently failing test cases based, in part, on a plurality of test results corresponding to each impacted test case of the plurality of impacted test cases,

wherein the plurality of test results comprises a failure count and a measure of the utilization of a resource of a computer system during execution of the corresponding impacted test case, and

wherein the prioritization is further based on a first pre-determined weighting factor of the measure of the utilization and a second pre-determined weighting factor of the failure count;

executing, based on the prioritization, one or more impacted test cases of the plurality of impacted test cases to obtain a plurality of executed test cases; and

obtaining, for each executed test case of the plurality of executed test cases, a new plurality of test results.

14. The non-transitory computer readable medium of claim 13 , wherein the method further comprises:

storing, in a database, the mapping and a plurality of test results corresponding to each test case of the plurality of test cases; and

updating, for each executed test case, the database with the new plurality of test results.

15. The non-transitory computer readable medium of claim 13 ,

wherein each test result of the plurality of test results comprises a failure count and an execution time, and

wherein the prioritization is further based on a third pre-determined weighting factor of the execution time.

16. The non-transitory computer readable medium of claim 13 , wherein the determination is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases.

17. The non-transitory computer readable medium of claim 13 ,

wherein the prioritization is further based on a precision of coverage, relative to a change in the change list, of an impacted test case of the plurality of impacted test cases, and a scope of a change in the change list.

18. The non-transitory computer readable medium of claim 13 , wherein the method further comprises:

issuing an alert identifying one or more lines in one or more files of the plurality of files based on one or more test results corresponding to a test case of the plurality of test cases.