Method and apparatus for recommending regression tests
Information associated with tests performed on an application program is captured and stored in memory. This information is then used to recommend a set of regression tests to a user for use in regression testing the application. Particularly, responsive to being notified about a change in the program code, the device analyzes the stored information and recommends a selected set of regression tests to a user based on the results of that analysis. The recommended tests enable the user to adequately regression test the program code in the application program using an optimal number of test cases.
1. A computer-implemented method comprising:
determining, by a processing circuit of a control computer, that program code in an application program has been modified;
determining, by the processing circuit, a set of test cases for the application program based on a mapping of the program code that has been modified to an execution trace of the application program stored in a memory circuit, wherein the execution trace was captured during a prior execution of the set of test cases and comprises a plurality of method identifiers, wherein each method identifier is associated with:
a method in a sequence of methods called by the program code prior to being modified;
a test case identifier identifying a test case that was executed to test the method during the prior execution of the set of test cases; and
a set of parameters associated with the test case and processed by the method during the execution of the test case identified by the test case identifier;
determining an optimum number of test cases that will be required for the regression testing based on a similarity between the parameters associated with the test cases in the set of test cases;
generating, by the processing circuit, a list of recommended test cases for regression testing the application program from the set of test cases to comprise the optimum number of test cases that will be required for the regression testing; and
identifying, by the processing circuit, the list of recommended test cases to a user.
2. The computer-implemented method of claim 1 wherein determining, by a processing circuit of a control computer, that program code in an application program has been modified comprises receiving an indication identifying which methods in the program code were modified.
3. The computer-implemented method of claim 2 wherein determining, by the processing circuit, a set of test cases for the application program based on a mapping of the program code that has been modified to an execution trace of the application program stored in a memory circuit comprises:
determining whether a method that was modified is in the sequence of methods of the execution trace; and
if the method is in the sequence of methods, including the test case identifier associated with the sequence of methods in the set of test cases.
4. The computer-implemented method of claim 1 further comprising:
prioritizing each test case in the set of test cases according to a predefined prioritization policy; and
generating the list of recommended test cases for regression testing the application program based on a priority of each test case.
5. The computer-implemented method of claim 4 wherein prioritizing each test case in the set of test cases according to a predefined prioritization policy comprises prioritizing each test case according to a number of times the test case was invoked during the prior execution of the set of test cases.
6. The computer-implemented method of claim 1 further comprising:
calculating a performance cost of executing each test case in the set of test cases; and
wherein generating the list of recommended test cases for regression testing the application program comprises adding test cases to, or deleting test cases from, the list of recommended test cases based on the calculated performance cost.
7. The computer-implemented method of claim 1 wherein identifying, by the processing circuit, the list of recommended test cases to a user comprises identifying a test script for each test case in the list of recommended test cases to the user.
8. The computer-implemented method of claim 1 wherein generating, by the processing circuit, a list of recommended test cases for regression testing the application program from the set of test cases comprises generating the list of recommended test cases based on a priority assigned to each test case, and on a calculated performance cost of executing each test case.
9. A computing device comprising:
a communications interface circuit configured to communicate data with a remote computing device via a communications network; and
a processing circuit configured to:
determine that program code in an application program has been modified;
determine a set of test cases for the application program based on a mapping of the program code that has been modified to an execution trace of the application program stored in a memory circuit, wherein the execution trace was captured during a prior execution of the set of test cases and comprises a plurality of method identifiers, wherein each method identifier is associated with:
a method in a sequence of methods called by the program code prior to being modified;
a test case identifier identifying a test case that was executed during the prior execution of the set of test cases; and
a set of parameters processed by the method during the execution of the test case identified by the test case identifier;
determine an optimum number of test cases that will be required for the regression testing based on a similarity between the parameters associated with the test cases in the set of test cases;
generate a list of recommended test cases for regression testing the application program from the set of test cases to comprise the optimum number of test cases that will be required for the regression testing; and
identify the list of recommended test cases to a user.
10. The computing device of claim 9 wherein the processing circuit is further configured to receive an indication identifying which methods in the program code were modified.
11. The computing device of claim 10 wherein the processing circuit is further configured to:
determine whether a method that was modified is in the sequence of methods of the execution trace; and
if the method is in the sequence of methods, include the test case identifier associated with the sequence of methods in the set of test cases.
12. The computing device of claim 9 wherein the processing circuit is further configured to:
prioritize each test case in the set of test cases according to a predefined prioritization policy; and
generate the list of recommended test cases for regression testing the application program based on a priority of each test case.
13. The computing device of claim 12 wherein the processing circuit is further configured to prioritize each test case according to a number of times the test case was invoked during the prior execution of the set of test cases.
14. The computing device of claim 9 wherein the processing circuit is further configured to:
calculate a performance cost of executing each test case in the set of test cases; and
add test cases to, or delete test cases from, the list of recommended test cases based on the calculated performance cost.
15. The computing device of claim 9 wherein the processing circuit is further configured to generate the list of recommended test cases based on a priority assigned to each test case, and on a calculated performance cost of executing each test case.
16. A non-transitory computer-readable storage medium comprising executable code stored thereon that, when executed by a processing circuit of a computing device, configures the computing device to:
determine that program code in an application program has been modified;
determine a set of test cases for the application program based on a mapping of the program code that has been modified to an execution trace of the application program stored in a memory circuit, wherein the execution trace was captured during a prior execution of the set of test cases and comprises a plurality of method identifiers, wherein each method identifier is associated with:
a method in a sequence of methods called by the program code prior to being modified;
a test case identifier identifying a test case that was executed during the prior execution of the set of test cases; and
a set of parameters processed by the method during the execution of the test case identified by the test case identifier;
determining an optimum number of test cases that will be required for the regression testing based on a similarity between the parameters associated with the test cases in the set of test cases;
generate a list of recommended test cases for regression testing the application program from the set of test cases to comprise the optimum number of test cases that will be required for the regression testing; and
identify the list of recommended test cases to a user.
17. The computer-implemented method of claim 8 wherein generating the list of recommended test cases based on a priority assigned to each test case, and on a calculated performance cost of executing each test case, comprises generating the list of recommended test cases to include:
test cases having a hit-rate that meets or exceeds a predetermined hit-rate threshold; and
test cases having a calculated performance cost that is lower than a predetermined performance cost threshold.