Method, program, and system for generating test cases
To provide a technique for generating, at a high speed, a smaller-sized set that satisfies an intended property such as, for example, being pair-wise, and includes many test cases that match a set of existing test cases given as an input, candidates to be used from a set of existing input test cases are determined in the following manner: for some parameters, values to be held by test case candidates are determined; test cases having the determined values, among those included in the set of input test cases, are selected as the candidates. A test case having the highest score among one or more test case candidates generated with the method of the related art and one or more test case candidates selected from the set of input test cases is added to a set of output test cases.
1. A computer-executed program product for generating test cases for a system having a plurality of parameters and one or more variable values assigned to each of the plurality of parameters, in accordance with combinations of the parameters and the variable values assigned to the parameters, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a computer to perform a method comprising:
holding a set of test cases including existing input test cases in storage means in a non-transitory computer readable manner, wherein a validity of each of the existing input test cases has previously been verified;
executing a loop until the combinations of the parameters and the variable values are completed for all of the plurality of parameters, wherein the loop comprises:
(i) generating the combinations of the parameters and the variable values so that the number of parameters storing variable values gradually increases,
(ii) finding a test case that contains combinations of parameters and variable values partially matching the generated combinations of the parameters and the variables in the set of test cases including the existing input test cases, and storing the found test case in a set variable, and
(iii) adding, in response to the set variable becoming an empty set, the test case that has been stored in the set variable in the last loop to a set of candidates;
adding, in response to the combinations of the parameters and the variable values being completed for all of the plurality of parameters, a test case generated on the basis of the completed combinations of the parameters and the variables to the set of candidates;
selecting, once the test cases for a plurality of different combinations of the parameters and the variable values are accumulated in the set of candidates, a test case from the set of candidates in accordance with the magnitude of a score related to a value including the number of cases that have not appeared,
determining that the selected test case is an existing input test case from the set of test cases and calculating the score of the selected test case by increasing the score of the selected test case by multiplying the score by a value greater than one,
otherwise, determining that the selected test case is not an existing input test case from the set of test cases and calculating the score for the selected test case without multiplying the score by any value;
outputting the selected test case from the set of candidates; and
testing the system by performing the selected test case by the system.
2. The computer-executed program product according to claim 1 , wherein the plurality of different combinations of the parameters and the variable values are generated by using a pair-wise method.
3. The computer-executed program product according to claim 2 , wherein the score of the test case is determined by the number of pairs that have not appeared among pairs covered by the test case.
4. A computer-implemented system having a plurality of parameters and one or more variable values assigned to each of the plurality of parameters and for generating test cases for the system in accordance with combinations of the parameters and the variable values assigned to the parameters, the computer-implemented system comprising:
a memory;
a processing device coupled to the memory;
a test case holding unit to hold a set of test cases including existing input test cases in a non-transitory computer readable manner, wherein a validity of each of the existing input test cases has previously been verified;
a loop executing unit to execute a loop until the combinations of the parameters and the variable values are completed for all of the plurality of parameters, wherein the loop comprises:
(i) generating the combinations of the parameters and the variable values so that the number of parameters storing variable values gradually increases,
(ii) finding a test case that contains combinations of parameters and variable values partially matching the generated combinations of the parameters and the variables in the set of test cases including the existing input test cases, and storing the found test case in a set variable, and
(iii) adding, in response to the set variable becoming an empty set, the test case that has been stored in the set variable in the last loop to a set of candidates;
a test case adding unit to add, in response to the combinations of the parameters and the variable values being completed for all of the plurality of parameters, a test case generated on the basis of the completed combinations of the parameters and the variables to the set of candidates;
a test case selecting unit to select, once the test cases for a plurality of different combinations of the parameters and the variable values are accumulated in the set of candidates, a test case from the set of candidates in accordance with the magnitude of a score related to a value including the number of cases that have not appeared,
determining that the selected test case is an existing input test case from the set of test cases and calculating the score of the selected test case by increasing the score by multiplying the score by a value greater than one;
otherwise, determining that the selected test case is not an existing input test case from the set of test cases and calculating the score for the selected test case without multiplying the score by any value; and
a test case output unit to output the selected test case from the set of candidates and to test the system by performing the selected test case by the system.
5. The computer-implemented system according to claim 4 , wherein the plurality of different combinations of the parameters and the variable values are generated by using a pair-wise method.
6. The computer-implemented system according to claim 5 , wherein the score of the test case is determined by the number of pairs that have not appeared among pairs covered by the test case.
7. A computer-executed program product for generating test cases for a system having a plurality of parameters and one or more variable values assigned to each of the plurality of parameters, in accordance with combinations of the parameters and the variable values assigned to the parameters, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by a computer to perform a method comprising:
holding a set of test cases, in a storage, including existing input test cases in a non-transitory computer readable manner, wherein a validity of each of the existing input test cases has previously been verified;
generating the combinations of the parameters and the variable values;
finding a test case that contains the combinations of the parameters and the variable values in the set of test cases including the existing input test cases, and storing the found test case in a set variable;
adding a test case generated on the basis of the combinations of the parameters and the variables to a set of candidates;
selecting, once the test cases for a plurality of different combinations of the parameters and the variable values are accumulated in the set of candidates, a test case from the set of candidates in accordance with the magnitude of a score related to a value including the number of cases that have not appeared,
determining that the selected test case is an existing input test case from the set of test cases and calculating the score of the selected test case by increasing the score of the selected test case by multiplying the score by a value greater than one;
otherwise, determining that the selected test case is not an existing input test case from the set of test cases and calculating the score for the selected test case without multiplying the score by any value;
outputting the selected test case from the set of candidates; and
testing the system by performing the selected test case by the system.
8. The computer-executed program product according to claim 7 , wherein the plurality of different combinations of the parameters and the variable values are generated by using a pair-wise method.
9. The computer-executed program product according to claim 8 , wherein the score of the test case is determined by the number of pairs that have not appeared among pairs covered by the test case.