Smart test case execution cycle assignment mechanism
Methods, system, and non-transitory processor-readable storage medium for feature coverage system are provided herein. An example method includes selecting a test case from a plurality of test cases in a software testing lifecycle system. A feature coverage system calculates a weighted metric associated with a test cycle for each test cycle in a plurality of test cycles, where each of the plurality of test cycles comprises a subset of the plurality of test cases. Using the weighted metric, the feature coverage system determines a test cycle in which to assign the test case to balance test feature coverage within the plurality of test cycles, and assigns the test case to the test cycle. The software testing lifecycle system then executes the test cycle on a system.
1. A method, comprising:
selecting a test case from a plurality of test cases in a software testing lifecycle system; calculating, by a feature coverage system, a weighted metric associated with a test cycle for each test cycle in a plurality of test cycles,
wherein each of the plurality of test cycles comprises a subset of the plurality of test cases;
using the weighted metric, determining, by the feature coverage system, a test cycle in which to assign the test case to balance test feature coverage within the plurality of test cycles;
assigning, by the feature coverage system, the test case to the test cycle; and
executing the test cycle on a system; and
evaluating at least one of test case distribution imbalance rate, tester system workload imbalance rate and feature coverage imbalance rate to assess test case assignment performance of the feature coverage system,
wherein the method is implemented by at least one processing device comprising a processor coupled to a memory;
repeating the steps of selecting, calculating, determining, and assigning for the each test case that is not assigned.
2. The method of claim 1 further comprising:
detecting a test failure that occurred during the execution of the test cycle on the system; and
defining the plurality of test cases based on the test failure.
3. The method of claim 1 further comprising:
for each test case in the plurality of test cases:
determining whether the each test case is assigned to any test cycle in the plurality of test cycles.
4. The method of claim 1 wherein selecting the test case from the plurality of test cases comprises:
determining the test case has a dependency on a second test case;
determining the second test case is not assigned to any test cycle in the plurality of test cycles; and
selecting the second test case instead of the test case.
5. The method of claim 1 wherein selecting the test case from the plurality of test cases comprises:
determining the test case has a dependency on a second test case;
determining the second test case is assigned to a second test cycle in the plurality of test cycles, wherein the plurality of test cycles has an order of execution starting from a first test cycle through a last test cycle; and
modifying the plurality of test cycles to remove the second test cycle and any test cycles that are scheduled to execute prior to the second test cycle according to the order of execution, from the plurality of test cycles when determining the weighted metric associated with the test cycle for each test cycle in the plurality of test cycles.
6. The method of claim 1 wherein determining, by the feature coverage system, the test cycle in which to assign the test case comprises:
selecting the test cycle with a minimum weighted metric from the plurality of test cycles.
7. The method of claim 6 wherein selecting the test cycle with the minimum weighted metric from the plurality of test cycles comprises:
identifying a subset of the plurality of test cycles, wherein each of the subset has the minimum weighted metric; and
selecting one of the subset based on a priority of execution associated with the test case.
8. The method of claim 7 wherein selecting one of the subset based on the priority of execution associated with the test case comprises:
identifying the priority of execution as a highest priority; and
selecting an earliest executing test cycle in the plurality of test cycles as the one of the subset, wherein the plurality of test cycles has an order of execution starting from a first test cycle through a last test cycle.
9. The method of claim 7 wherein selecting one of the subset based on the priority of execution associated with the test case comprises:
identifying the priority of execution as a medium priority; and
randomly selecting a random test cycle in the plurality of test cycles as the one of the subset.
10. The method of claim 7 wherein selecting one of the subset based on the priority of execution associated with the test case comprises:
identifying the priority of execution as a lowest priority; and
selecting a last test cycle in the plurality of test cycles as the one of the subset, wherein the plurality of test cycles has an order of execution starting from a first test cycle through the last test cycle.
11. The method of claim 1 wherein each test case within a test cycle comprises an attribute set comprising at least one of a test case tag identifying features tested by the test case, test case execution time, tester system responsible for invoking execution of the test case, priority of execution of the test case, prior case dependency associated with the test case and test cycle to which the test case is assigned.
12. The method of claim 1 wherein calculating, by the feature coverage system, the weighted metric comprises:
determining an extent to which a test case feature is covered within a test case.
13. The method of claim 1 wherein calculating, by the feature coverage system, the weighted metric comprises:
determining an extent to which a test case feature is covered within a test cycle.
14. The method of claim 1 wherein calculating, by the feature coverage system, the weighted metric comprises:
determining a test case feature coverage metric comprising an extent to which a plurality of test case features associated with a test case is covered within a test cycle.
15. The method of claim 1 wherein calculating, by the feature coverage system, the weighted metric comprises:
determining a test case workload metric associated with a tester system invoking execution of the test cycle, wherein the test case workload metric is comprised of a total execution time of test cases within the test cycle and a predefined time period during which the tester system executes the test cycle.
16. The method of claim 1 wherein the weighted metric comprises a test case feature coverage metric, and a test case workload metric.
17. The method of claim 16 wherein the weighted metric comprises weighting each of the test case feature coverage metric and the test case workload metric.
18. A system comprising:
at least one processing device comprising a processor coupled to a memory; the at least one processing device being configured:
to select a test case from a plurality of test cases in a software testing lifecycle system;
to calculate, by a feature coverage system,
a weighted metric associated with a test cycle for each test cycle in a plurality of test cycles, wherein each of the plurality of test cycles comprises a subset of the plurality of test cases;
using the weighted metric, to determine, by the feature coverage system, a test cycle in which to assign the test case to balance test feature coverage within the plurality of test cycles;
to assign, by the feature coverage system, the test case to the test cycle; and
to execute the test cycle on a system; and
to evaluate at least one of test case distribution imbalance rate, tester system workload imbalance rate and feature coverage imbalance rate to assess test case assignment performance of the feature coverage system;
repeating the steps of selecting, calculating, determining, and assigning for the each test case that is not assigned.
19. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to select a test case from a plurality of test cases in a software testing lifecycle system;
to calculate, by a feature coverage system, a weighted metric associated with a test cycle for each test cycle in a plurality of test cycles, wherein each of the plurality of test cycles comprises a subset of the plurality of test cases;
using the weighted metric, to determine, by the feature coverage system, a test cycle in which to assign the test case to balance test feature coverage within the plurality of test cycles;
to assign, by the feature coverage system, the test case to the test cycle; and
to execute the test cycle on a system; and
to evaluate at least one of test case distribution imbalance rate, tester system workload imbalance rate and feature coverage imbalance rate to assess test case assignment performance of the feature coverage system;
repeating the steps of selecting, calculating, determining, and assigning for the each test case that is not assigned.