Automatic evaluation of test code quality
Techniques and solutions are described for automatically evaluating test code. In one technique, test code quality is evaluated by comparing assertions in test code with output values in target code tested by the test code. Output values that are not associated with assertions, or an insufficient number or variety of assertions can indicate that a test can be improved. In another technique, test quality is assessed by dynamically changing target code or test data used with a test. Room for test improvement can be indicated if test code provides a passing result despite changes to test data used with the test or changes to target code executed in conducting the test.
1. A method, implemented in a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising:
receiving a request to analyze test code;
retrieving the test code and target code tested by the test code;
determining one or more input parameters in the target code;
determining one or more output parameters in the target code, wherein a given output parameter of the one or more output parameter also serves as an input parameter or serves only as an output parameter;
determining if the test code comprises assertions for the one or more output parameters, wherein an assertion specifies a predicate for an output parameter and execution of the test code returns a Boolean value for the predicate associated with a given assertion;
determining an amount of operations in the target code that process a first input parameter selected from the one or more input parameters;
determining a number of assertions in the test code for a first output parameter selected from the one or more output parameters and having a value determined at least in part using a value of the first input parameter;
comparing the number of assertions with a threshold; and
providing analysis results, wherein the analysis results are based at least in part on the amount of operations in the target code that process the first input parameter and whether the threshold is satisfied.
2. The method of claim 1 , further comprising:
assigning an importance value to the first output parameter or a second output parameter of the one or more output parameters based at least in part on the amount of operations in the target code that process the first input parameter.
3. The method of claim 2 , wherein the analysis results comprise a score, the score being determined at least in part using a weighting calculated at least in part using the importance value.
4. The method of claim 1 , further comprising:
determining a score based at least in part on a number of output parameters of the one or more output parameters not associated with an assertion.
5. A computing system comprising:
at least one memory;
one or more hardware processing units coupled to the at least one memory; and
one or more computer readable storage media storing instructions that, when executed, cause the computing system to perform operations comprising:
receiving a request to analyze test code;
retrieving the test code and target code tested by the test code;
determining one or more input parameters in the target code;
determining one or more output parameters in the target code, wherein a given output parameter of the one or more output parameter also serves as an input parameter or serves only as an output parameter;
determining if the test code comprises assertions for the one or more output parameters, wherein an assertion specifies a predicate for an output parameter and execution of the test code returns a Boolean value for the predicate associated with a given assertion;
determining an amount of operations in the target code that process a first input parameter selected from the one or more input parameters;
determining a number of assertions in the test code for a first output parameter selected from the one or more output parameters and having a value determined at least in part using a value of the first input parameter;
comparing the number of assertions with a threshold; and
providing analysis results, wherein the analysis results are based at least in part on the amount of operations in the target code that process the first input parameter and whether the threshold is satisfied.
6. The computing system of claim 5 , the operations further comprising:
assigning an importance value to the first output parameter or a second output parameter of the one or more output parameters based at least in part on the amount of operations in the target code that process the first input parameter.
7. The computing system of claim 6 , wherein the analysis results comprise a score, the score being determined at least in part using a weighting calculated at least in part using the importance value.
8. The computing system of claim 5 , wherein the operations comprise:
determining a score based at least in part on a number of output parameters of the one or more output parameters not associated with an assertion.
9. The computing system of claim 5 , the operations further comprising:
determining a first value specified in the test code for the first input parameter or a second input parameter selected from the one or more input parameters;
determining an assertion specified in the test code for the first output parameter;
executing the test code using a second value instead of the first value; and
determining if the assertion is valid based on execution of the test code using the second value.
10. The computing system of claim 9 , wherein the third parameter is associated with a data element of a data model, the data element being associated with a domain, the domain consisting of valid values for the data element, and wherein the second value is selected from the domain.
11. The computing system of claim 9 , further comprising:
(1) determining one or more values for the first input parameter used in one or more executions of a plurality of executions of the test code; and
selecting the second value from the one or more values; or
(2) randomly selecting the second value.
12. The computing system of claim 11 , further comprising:
training a machine learning algorithm with the one or more values to generate a machine learning model;
wherein selecting the second value from the one or more values is carried out using the machine learning model.
13. The computing system of claim 9 , further comprising, after determining if the assertion is valid:
determining that additional test code evaluation should be performed;
determining that the target code should be modified;
determining a portion of the target code that affects a value of the second output parameter; and
altering the portion of the target code.
14. The computing system of claim 13 , wherein the altering comprises commenting out at least one operation defined in the portion of the target code.
15. The computing system of claim 13 , wherein altering the portion of the target code is based at least in part on output of a machine learning model.
16. The computing system of claim 9 , wherein the test code is executed on the target code a plurality of times, and additional executions are performed until termination criteria are satisfied.
17. One or more non-transitory computer-readable storage media comprising:
computer-executable instructions that, when executed by a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to receive a request to analyze test code;
computer-executable instructions that, when executed by the computing system, cause the computing system to retrieve the test code and target code tested by the test code;
computer-executable instructions that, when executed by the computing system, cause the computing system to determine one or more input parameters in the target code;
computer-executable instructions that, when executed by the computing system, cause the computing system to determine one or more output parameters in the target code, wherein a given output parameter of the one or more output parameter also serves as an input parameter or serves only as an output parameter;
computer-executable instructions that, when executed by the computing system, cause the computing system to determine if the test code comprises assertions for the one or more output parameters, wherein an assertion specifies a predicate for an output parameter and execution of the test code returns a Boolean value for the predicate associated with a given assertion;
computer-executable instructions that, when executed by the computing system, cause the computing system to determine an amount of operations in the target code that process a first input parameter selected from the one or more input parameters;
computer-executable instructions that, when executed by the computing system, cause the computing system to determine a number of assertions in the test code for a first output parameter selected from the one or more output parameters and having a value determined at least in part using a value of the first input parameter;
computer-executable instructions that, when executed by the computing system, cause the computing system to compare the number of assertions with a threshold; and
computer-executable instructions that, when executed by the computing system, cause the computing system to provide analysis results, wherein the analysis results are based at least in part on the amount of operations in the target code that process the first input parameter and whether the threshold is satisfied.
18. The one or more non-transitory computer-readable storage media of claim 17 , further comprising:
computer-executable instructions that, when executed by the computing system, cause the computing system to assign an importance value to the first output parameter or a second output parameter of the one or more output parameters based at least in part on the amount of operations in the target code that process the first input parameter.
19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the analysis results comprise a score, the score being determined at least in part using a weighting calculated at least in part using the importance value.
20. The one or more non-transitory computer-readable storage media of claim 17 , further comprising:
computer-executable instructions that, when executed by the computing system, cause the computing system to determine a score based at least in part on a number of output parameters of the one or more output parameters not associated with an assertion.