Dynamic scheduling of test cases
Dynamically scheduled testing of a software application on one or more devices are disclosed. The system can schedule and execute of one or more test cases associated with a software application. When a device becomes available for testing, the system can determine the list of commits that have not yet been tested. If there are outstanding untested commits, the system can determine the list of test cases in a test plan that are to be scheduled and executed. For each test in the list of test cases, the system can compute an urgency value. Urgency value can be a function of the untested commits. For example, urgency value can be the number of untested commits associated with each test case. The system can then sort the test cases based on the urgency value and schedule the most “urgent” test case for execution.
1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for dynamically scheduling testing of test cases in a list of test cases, the operations comprising:
determining a list of untested commits, wherein each commit is associated with a modified copy of a source code file;
for each selected test case in the list of test cases, computing, by an urgency value computation engine, an urgency value of the selected test case such that the urgency value is initially computed as a number of untested commits associated with the selected test case;
sorting the test cases in the list of test cases in decreasing order of urgency value;
identifying a test case for execution from the list of test cases as the most urgent test case based on the computed urgency value of the test case being the highest urgency value from among urgency values computed for the list of test cases; and
executing the most urgent test case prior to executing a remainder of the test cases in the list of test cases.
2. The non-transitory computer-readable storage medium of claim 1 wherein the operations further comprise:
adjusting the urgency value of the selected test case based on a priority factor.
3. The non-transitory computer-readable storage medium of claim 1 wherein the operations further comprise:
adjusting the urgency value of the selected test case based on a number of commits since a previous execution of the selected test case.
4. The non-transitory computer-readable storage medium of claim 1 wherein the operations further comprise:
adjusting the urgency value of the selected test case based on a previous execution timestamp of the selected test case.
5. The non-transitory computer-readable storage medium of claim 1 wherein the operations further comprise:
associating a weight value with at least one untested commit in the list of untested commits, wherein the weight value is based on a probability of regression caused by the at least one untested commit; and
adjusting the urgency value of the selected test case based on weight values of untested commits associated with the selected test case.
6. The non-transitory computer-readable storage medium of claim 1 wherein the operations further comprise:
computing a wait time for each untested commit in the list of untested commits; and
adjusting the urgency value of the selected test case based on wait times of untested commits associated with the selected test case.
7. The non-transitory computer-readable storage medium of claim 1 wherein the operations further comprise:
logging the urgency value of each test case in the list of test cases.
8. A method performed by a computing system for dynamically scheduling testing of test cases in a list of test cases, comprising:
determining a list of untested commits, wherein each commit is associated with a modified copy of a source code file;
for each selected test case in the list of test cases, computing, by an urgency value computation engine, an urgency value of the selected test case such that the urgency value is initially computed as a number of untested commits associated with the selected test case;
sorting the test cases in the list of test cases in decreasing order of urgency value;
identifying a test case for execution from the list of test cases as the most urgent test case based on the computed urgency value of the test case being the highest urgency value from among urgency values computed for the list of test cases; and
executing the most urgent test case prior to executing a remainder of the test cases in the list of test cases.
9. The method of claim 8 , further comprising:
adjusting the urgency value of the selected test case based on a priority factor.
10. The method of claim 8 , further comprising:
adjusting the urgency value of the selected test case based on a number of commits since a previous execution of the selected test case.
11. The method of claim 8 , further comprising:
adjusting the urgency value of the selected test case based on a previous execution timestamp of the selected test case.
12. The method of claim 8 , further comprising:
associating a weight value with at least one untested commit in the list of untested commits, wherein the weight value is based on a probability of regression caused by the at least one untested commit; and
adjusting the urgency value of the selected test case based on weight values of untested commits associated with the selected test case.
13. The method of claim 8 , further comprising:
computing a wait time for each untested commit in the list of untested commits; and
adjusting the urgency value of the selected test case based on wait times of untested commits associated with the selected test case.
14. The method of claim 8 , further comprising:
logging the urgency value of each test case in the list of test cases.
15. A system for dynamically scheduling testing of test cases in a list of test cases, comprising:
at least one hardware processor coupled to a network; and
a memory configured to store processor-executable components, wherein the processor-executable components further comprise:
a component configured to determine a list of untested commits, wherein each commit is associated with a modified copy of a source code file;
a component comprising an urgency value computation engine to compute, for each selected test case in the list of test cases, an urgency value of the selected test case such that the urgency value is initially computed as a number of untested commits associated with the selected test case;
a component to sort the test cases in the list of test cases in decreasing order of urgency value;
a component to adjust the urgency value of the selected test case based on a modification factor;
a component configured to identify a test case for execution from the list of test cases as the most urgent test case based on the computed urgency value of the test case being the highest urgency value from among urgency values computed for the list of test cases; and
a component to execute the most urgent test case prior to executing a remainder of the test cases in the list of test cases.
16. The system of claim 15 wherein the modification factor is based on a priority value associated with the selected test case.
17. The system of claim 15 wherein the modification factor is based on a number of commits since a previous execution of the selected test case.
18. The system of claim 15 wherein the modification factor is based on a previous execution timestamp of the selected test case.
19. The system of claim 15 wherein the processor-executable components further comprise a component configured to:
associate a weight value with at least one untested commit in the list of untested commits, wherein the weight value is based on a probability of regression caused by the at least one untested commit; and
adjust the urgency value of the selected test case based on weight values of untested commits associated with the selected test case.
20. The system of claim 15 wherein the processor-executable components further comprise a component configured to:
compute a wait time for each untested commit in the list of untested commits; and
adjust the urgency value of the selected test case based on wait times of untested commits associated with the selected test case.