Test coverage optimizing mechanism based on metric evaluation system

1 . A method, comprising:

executing at least one test case on a test system under controlled test conditions;

automatically collecting test execution data during said execution including test related failures and system performance metrics;

calculating, by a weighted average baseline metrics calculation module implemented in the information processing system, a weighted average baseline metric for the at least one test case using the test related failures that occurred during the execution;

calculating, by a coverage imbalance module implemented in the information processing system, a coverage imbalance associated with a test domain using the weighted average baseline metric, wherein the test domain comprises the at least one test case, wherein the coverage imbalance module calculates each evaluation metric's coverage imbalance degree for the test domain from a plurality of evaluation metrics comprising diversity, endurance, performance, availability, and scalability, wherein the plurality of evaluation metrics are interdependently weighted and evaluated together to determine overall test coverage balance for improving test system reliability, wherein:

diversity represents a count of object types and input/output types tested in the test system;

endurance represents runtime duration of the test case executing on the test system;

performance represents load status measured by input/output operations performed by the test system;

availability represents a count of errors and faults injected into the test system; and

scalability represents a number of objects created in the test system during test execution; and

automatically generating test system improvement recommendations based on the calculated coverage imbalance, wherein the method is implemented by at least one processing device comprising a processor coupled to a memory and configured to interface with the test system.

2 . The method of claim 1 further comprising:

determining the coverage imbalance associated with the test domain at least meets a coverage imbalance threshold.

3 . The method of claim 2 wherein determining the coverage imbalance associated with the test domain at least meets the coverage imbalance threshold comprises:

identifying at least one evaluation metric that requires improvement.

4 . The method of claim 2 wherein determining the coverage imbalance associated with the test domain at least meets the coverage imbalance threshold comprises:

identifying, by a test improvement module, at least one test case in the test domain that requires improvement.

5 . The method of claim 2 wherein determining the coverage imbalance associated with the test domain at least meets the coverage imbalance threshold comprises:

classifying, by a test domain classification module, the test domain according to the coverage imbalance associated with the test domain and the test related failures that occurred during the execution.

6 . The method of claim 5 further comprising:

determining a validity associated with the test domain based on at least one of the coverage imbalance and a number of test related failures that occurred during the execution as compared to a minimum average bug count per test case.

7 . The method of claim 5 further comprising:

determining an evaluation metric associated with the test domain that requires improvement based on the classification of the test domain.

8 . The method of claim 1 further comprising:

defining at least one evaluation metric for at least one test case.

9 . The method of claim 8 further comprising:

calculating a coverage imbalance degree for the at least one evaluation metric for the at least one test case.

10 . The method of claim 1 wherein the at least one test case comprises

an attribute set according to the plurality of evaluation metrics.

11 . The method of claim 1 wherein the coverage imbalance module identifies at least one of a test case and the at least one evaluation metric that requires improvement to achieve a desired coverage imbalance.

12 . The method of claim 1 wherein calculating, by the weighted average baseline metrics calculation module, the weighted average baseline metric for the at least one test case comprises:

calculating a weight associated with the at least one test case based on a number of the test related failures that occurred during execution.

13 . The method of claim 1 wherein calculating, by the weighted average baseline metrics calculation module, the weighted average baseline metric for the at least one test case comprises:

calculating an average baseline for at least one evaluation metric associated with the at least one test case.

14 . The method of claim 1 wherein calculating, by the weighted average baseline metrics calculation module, the weighted average baseline metric for the at least one test case comprises:

calculating an average baseline for at least one evaluation metric associated with the test domain.

15 . The method of claim 1 wherein the test domain comprises testing at least one of a component associated with the system and a feature executing on the system.

16 . A system comprising:

at least one processing device comprising a processor coupled to a memory;

the at least one processing device being configured:

to execute at least one test case on a test system under controlled test conditions;

to automatically collect test execution data during said execution including test related failures and system performance metrics;

to calculate, by a weighted average baseline metrics calculation module implemented in the information processing system, a weighted average baseline metric for the at least one test case using the test related failures that occurred during the execution;

to calculate, by a coverage imbalance module implemented in the information processing system, a coverage imbalance associated with a test domain using the weighted average baseline metric, wherein the test domain comprises the at least one test case, wherein the coverage imbalance module calculates each evaluation metric's coverage imbalance degree for the test domain from a plurality of evaluation metrics comprising diversity, endurance, performance, availability, and scalability, wherein the plurality of evaluation metrics are interdependently weighted and evaluated together to determine overall test coverage balance for improving test system reliability, wherein:

diversity represents a count of object types and input/output types tested in the test system;

endurance represents runtime duration of the test case executing on the test system;

performance represents load status measured by input/output operations performed by the test system;

availability represents a count of errors and faults injected into the test system; and

scalability represents a number of objects created in the test system during test execution; and

automatically generate test system improvement recommendations based on the calculated coverage imbalance.

17 . The system of claim 16 further configured:

to define at least one evaluation metric for at least one test case.

18 . The system of claim 17 further configured:

to calculate a coverage imbalance degree for the at least one evaluation metric for the at least one test case.

19 . The system of claim 17 further configured:

to calculate a coverage imbalance degree for the at least one evaluation metric for the test domain.

20 . A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to execute at least one test case on a test system under controlled test conditions;

to automatically collect test execution data during said execution including test related failures and system performance metrics;

to calculate, by a weighted average baseline metrics calculation module implemented in the information processing system, a weighted average baseline metric for the at least one test case using the test related failures that occurred during the execution;

to calculate, by a coverage imbalance module implemented in the information processing system, a coverage imbalance associated with a test domain using the weighted average baseline metric, wherein the test domain comprises the at least one test case, wherein the coverage imbalance module calculates each evaluation metric's coverage imbalance degree for the test domain from a plurality of evaluation metrics comprising diversity, endurance, performance, availability, and scalability, wherein the plurality of evaluation metrics are interdependently weighted and evaluated together to determine overall test coverage balance for improving test system reliability, wherein:

diversity represents a count of object types and input/output types tested in the test system;

endurance represents runtime duration of the test case executing on the test system;

performance represents load status measured by input/output operations performed by the test system;

availability represents a count of errors and faults injected into the test system; and

scalability represents a number of objects created in the test system during test execution; and

automatically generate test system improvement recommendations based on the calculated coverage imbalance.