Data-driven scheduling of automated software program test suites
Techniques are provided for data-driven scheduling of automated software program test suites. An exemplary method comprises: obtaining a plurality of test cases that test software programs; obtaining a failure likelihood for each of the plurality of test cases; and scheduling the test cases in an order based on the failure likelihoods. Generally, test cases that are more likely to fail are scheduled before test cases that are less likely to fail. Dependencies and/or priorities among the plurality of test cases are also optionally obtained and the scheduling of the test cases is based on the dependencies and/or priorities. The dependencies among the plurality of test cases comprise, for example, an indication of whether a given test case constrains one or more additional test cases.
1. A method, comprising:
obtaining a plurality of test cases that test one or more software programs;
obtaining dependencies among two or more of the plurality of test cases, wherein the dependencies among the two or more test cases comprise an indication of whether a given identified test case constrains one or more additional test cases;
obtaining a failure likelihood for each of the plurality of test cases, wherein the failure likelihood for at least one of the plurality of test cases is calculated using a Bayesian inference that calculates a generalized weighted probability for the at least one test case using a weight based at least in part on a number of prior executions of the at least one test case; and
scheduling, using at least one processing device, an execution of a plurality of the test cases in a particular order based at least in part on the failure likelihoods and the dependencies such that the given identified test case is executed before the one or more additional test cases, wherein the failure likelihoods and the dependencies are distinct from one another.
2. The method of claim 1 , wherein the failure likelihoods are based on historical test data.
3. The method of claim 1 , wherein the scheduling is further based on one or more of (i) test cases that were already scheduled and (ii) one or more portions of the one or more software programs that were modified.
4. The method of claim 1 , wherein test cases that are more likely to fail are scheduled before test cases that are less likely to fail.
5. The method of claim 1 , further comprising the step of assessing the failure likelihoods of the plurality of test cases using one or more machine learning techniques.
6. The method of claim 1 , further comprising obtaining priorities among two or more of the plurality of test cases from a user and wherein the scheduling is further based on the priorities.
7. The method of claim 1 , wherein the failure likelihoods are estimated using one or more parameters obtained by conducting a grid search over a dataset of prior test suites to find values of the one or more parameters that substantially minimize a run-time over the prior test suites.
8. The method of claim 1 , wherein the failure likelihoods are estimated using one or more parameters configured by a user.
9. A system, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
obtaining a plurality of test cases that test one or more software programs;
obtaining dependencies among two or more of the plurality of test cases, wherein the dependencies among the two or more test cases comprise an indication of whether a given identified test case constrains one or more additional test cases;
obtaining a failure likelihood for each of the plurality of test cases, wherein the failure likelihood for at least one of the plurality of test cases is calculated using a Bayesian inference that calculates a generalized weighted probability for the at least one test case using a weight based at least in part on a number of prior executions of the at least one test case; and
scheduling, using at least one processing device, an execution of a plurality of the test cases in a particular order based at least in part on the failure likelihoods and the dependencies such that the given identified test case is executed before the one or more additional test cases, wherein the failure likelihoods and the dependencies are distinct from one another.
10. The system of claim 9 , wherein test cases that are more likely to fail are scheduled before test cases that are less likely to fail.
11. The system of claim 9 , further comprising the step of assessing the failure likelihoods of the plurality of test cases using one or more machine learning techniques.
12. The system of claim 9 , further comprising obtaining priorities among two or more of the plurality of test cases from a user and wherein the scheduling is further based on the priorities.
13. The system of claim 9 , wherein the failure likelihoods are estimated using one or more parameters obtained by conducting a grid search over a dataset of prior test suites to find values of the one or more parameters that substantially minimize a run-time over the prior test suites.
14. The system of claim 9 , wherein the failure likelihoods are estimated using one or more parameters configured by a user.
15. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
obtaining a plurality of test cases that test one or more software programs;
obtaining dependencies among two or more of the plurality of test cases, wherein the dependencies among the two or more test cases comprise an indication of whether a given identified test case constrains one or more additional test cases;
obtaining a failure likelihood for each of the plurality of test cases, wherein the failure likelihood for at least one of the plurality of test cases is calculated using a Bayesian inference that calculates a generalized weighted probability for the at least one test case using a weight based at least in part on a number of prior executions of the at least one test case; and
scheduling, using at least one processing device, an execution of a plurality of the test cases in a particular order based at least in part on the failure likelihoods and the dependencies such that the given identified test case is executed before the one or more additional test cases, wherein the failure likelihoods and the dependencies are distinct from one another.
16. The computer program product of claim 15 , wherein test cases that are more likely to fail are scheduled before test cases that are less likely to fail.
17. The computer program product of claim 15 , further comprising the step of assessing the failure likelihoods of the plurality of test cases using one or more machine learning techniques.
18. The computer program product of claim 15 , further comprising obtaining priorities among two or more of the plurality of test cases from a user and wherein the scheduling is further based on the priorities.
19. The computer program product of claim 15 , wherein the failure likelihoods are estimated using one or more parameters obtained by conducting a grid search over a dataset of prior test suites to find values of the one or more parameters that substantially minimize a run-time over the prior test suites.
20. The computer program product of claim 15 , wherein the failure likelihoods are estimated using one or more parameters configured by a user.