Automatically rerunning test executions
Example implementations relate to automatically rerunning test executions. Some implementations may capture data during executions of a test. The data may include test status data, test rerun data, test owner data, and/or code committer data. Some implementations may also dynamically determine, for a failed execution of the test, a number of reruns to execute based on the captured data. Additionally, some implementations may cause in response to the dynamic determination, automatic rerun executions of the test until one of the rerun executions passes, the rerun executions being performed up to the number of times.
1. A system for automatically rerunning test executions comprising:
a processor; and
a memory storing instructions that when executed cause the processor to:
capture data during executions of an application-under-test (AUT);
determine, based on historical data of a failed test of the AUT, an average number of reruns of the failed test needed until the failed test passed;
determine, based on the historical data of the failed test, a highest number of reruns of the failed test needed until the failed test passed;
determine a rank value of a particular code committer who committed the AUT for testing;
determine whether the rank value of the particular code committer is within a first predetermined range;
in response to a determination that the rank value of the particular code committer is within the first predetermined range, cause the failed test to automatically rerun until the failed test passes, wherein the failed test is rerun up to the determined average number of reruns;
in response to a determination that the ranking value of the particular code committer is not within the first predetermined range, cause the failed test to automatically rerun until the failed test passes, wherein the failed test is rerun up to the determined highest number of reruns; and
in response to a determination that the failed test does not pass after the determined average number of reruns or after the determined highest number of reruns, generate an alert of the failed test to the particular code committer.
2. The system of claim 1 , wherein the alert causes the particular code committer to determine whether the failed test was due to a real application failure or due to a test instability.
3. The system of claim 1 , wherein the determination of the rank value of the particular code committer is based on a comparison between a performance of the particular code committer and performances of other code committers.
4. The system of claim 1 , wherein the determination of the rank value of the particular code committer is based on a performance ratio between a total number of tests failed by the particular code committer and a total number of tests associated with the particular code committer.
5. The system of claim 1 , wherein, prior to the determination of whether the rank value of the particular code committer is within the first predetermined range, the instructions are executable to cause the processor to:
determine, based on the historical data, a test owner value of a particular test owner who wrote the failed test; and
determine a total number of reruns of the failed test to execute based on the test owner value of the particular test owner.
6. The system of claim 5 , wherein the instructions are executable to cause the processor to:
determine whether the test owner value is within a second predetermined range corresponding to a poor owner performance;
in response to a determination that the test owner value is within the second predetermined range, assign a predetermined value as the total number of reruns; and
in response to a determination that the test owner value is not within the second predetermined range, determine whether the rank value of the particular code committer is within the first predetermined range.
7. The system of claim 5 , wherein, to determine the test owner value of the particular test owner, the instructions are executable to cause the processor to:
determine a test owner performance ratio between a total number of failed tests written or changed by the particular test owner and a total number of tests written or changed by the particular test owner; and
calculate the test owner value of the particular test owner based on the test owner performance ratio.
8. The system of claim 1 , wherein:
the captured data further includes data related to whether an execution of the AUT passed or failed, data related to whether the execution of the AUT is an original execution of the AUT or a rerun execution of the AUT, data related to an individual that wrote or changed the failed test, and data related to the particular code committer.
9. A computer-implemented method comprising:
capturing, by a processor of a computing device, data during executions of an application-under-test (AUT);
determining, based on historical data of a failed test of the AUT, an average number of reruns of the failed test needed until the failed test passed;
determining, based on the historical data of the failed test, a highest number of reruns of the failed test needed until the failed test passed;
determining, by the processor, a rank value of a particular code committer who committed the AUT for testing;
determining, by the processor, whether the rank value of the particular code committer is within a first predetermined range;
in response to a determination that the rank value of the particular code committer is within the first predetermined range, causing, by the processor, the failed test to automatically rerun until the failed test passes, wherein the failed test is rerun up to the determined average number of reruns;
in response to a determination that the ranking value of the particular code committer is not within the first predetermined range, causing, by the processor, the failed test to automatically rerun until the failed test passes, wherein the failed test is rerun up to the determined highest number of reruns; and
in response to a determination that the failed test does not pass after the determined average number of reruns or after the determined highest number of reruns, generating, by the processor, an alert of the failed test to the particular code committer.
10. The computer-implemented method of claim 9 , wherein the method comprises, prior to determining whether the rank value of the particular code committer is within the first predetermined range:
determining a test owner value of a particular test owner who wrote or changed the failed test, and
determining a total number of reruns of the failed test to execute based on the test owner value of the particular test owner.
11. The computer-implemented method of claim 10 , wherein the test owner value of the particular test owner is determined based on a comparison between a performance of the particular test owner and performances of other test owners.
12. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to:
capture data during executions of an application-under-test (AUT);
determine, based on historical data of a failed test of the AUT, an average number of reruns of the failed test needed until the failed test passed;
determine, based on the historical data of the failed test, a highest number of reruns of the failed test needed until the failed test passed;
determine a rank value of a particular code committer who committed the AUT for testing;
determine whether the rank value of the particular code committer is within a first predetermined range;
in response to a determination that the rank value of the particular code committer is within the first predetermined range, cause the failed test to automatically rerun until the failed test passes, wherein the failed test is rerun up to the determined average number of reruns;
in response to a determination that the ranking value of the particular code committer is not within the first predetermined range, cause the failed test to automatically rerun until the failed test passes, wherein the failed test is rerun up to the determined highest number of reruns; and
in response to a determination that the failed test does not pass after the determined average number of reruns or after the determined highest number of reruns, generate an alert of the failed test to the particular code committer.
13. The non-transitory computer-readable storage medium of claim 12 , wherein, prior to the determination of whether the rank value of the particular code committer is within the first predetermined range, the instructions are executable to cause the processor to:
determine a test owner value of a particular test owner who wrote the failed test, and
determine a total number of reruns of the failed test to execute based on the test owner value of the particular test owner.
14. The non-transitory computer-readable storage medium of claim 13 , wherein, to determine the test owner value, the instructions are executable to cause the processor to:
determine, based on the historical data, a total number of failed tests written or changed by the particular test owner and a total number of tests written or changed by the particular test owner; and
calculate the test owner value based on a ratio between the total number of failed tests written or changed by the particular test owner and the total number of tests written or changed by the particular test owner.
15. The non-transitory computer-readable storage medium of claim 12 , wherein, to determine the rank value of the particular code committer, the instructions are executable to cause the processor to:
determine, based on the historical data, a committer performance ratio between a total number of tests failed by the particular code committer and a total number of tests associated with the particular code committer; and
calculate the rank value of the particular code committer based on the committer performance ratio.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions are executable to cause the processor to:
determine whether the test owner value is within a second predetermined range corresponding to a poor owner performance;
in response to a determination that the test owner value is within the second predetermined range, assign a predetermined value as the total number of reruns; and
in response to a determination that the test owner value is not within the second predetermined range, determine whether the rank value of the particular code committer is within the first predetermined range.
17. The computer-implemented method of claim 9 , wherein determining the rank value of the particular code committer value comprises:
determining, based on the historical data, a committer performance ratio between a total number of tests failed by the particular code committer and a total number of tests associated with the particular code committer; and
calculating the rank value of the particular code committer based on the committer performance ratio.
18. The computer-implemented method of claim 10 , wherein determining the total number of reruns of the failed test to execute comprises:
determining whether the test owner value is within a second predetermined range corresponding to a poor owner performance;
in response to a determination that the test owner value is within the second predetermined range, assigning a predetermined value as the total number of reruns; and
in response to a determination that the test owner value is not within the second predetermined range, determining whether the rank value of the particular code committer is within the first predetermined range.
19. The computer-implemented method of claim 9 , wherein the rank value of the particular code committer is determined based on a comparison between a performance of the particular code committer and performances of other code committers.
20. The non-transitory computer-readable storage medium of claim 12 , wherein the rank value of the particular code committer is determined based on a comparison between a performance of the particular code committer and performances of other code committers.