Reliable automatic estimation of test flakiness

1. A method, comprising:

retrieving, from a repository, a software test and a software module;

executing the software test against the software module a first number of times, observing a first number of failures, to obtain a first flakiness value;

executing the software test against the software module a second number of times, observing a second number of failures, to obtain a second flakiness value;

in response to a convergence criterion between the first flakiness value and the second flakiness value not being met, executing the software test against the software module a third number of times, observing a third flakiness value; and

in response to a difference between one of the first flakiness value, the second flakiness value, or the third flakiness value and a set of historical flakiness values exceeding a threshold, creating a defect record.

2. The method of claim 1 , wherein the executing the software test is responsive to a change in the software module.

3. The method of claim 1 , wherein the threshold is at least 0.001.

4. The method of claim 1 , wherein the executing the software test further comprises a frequency of executing the software test between 5 and 30 days.

5. The method of claim 1 , wherein at least one of the first flakiness value, the second flakiness value, or the third flakiness value are determined using at least one of:

the convergence criterion;

a Student's t-test; or

an error propagation.

6. The method of claim 5 , wherein the error propagation comprises a standard deviation and a stopping criterion.

7. The method of claim 1 , wherein the executing the software test further comprises executing the software test a fourth number of times, periodically, at a first frequency interval.

8. The method of claim 1 , further comprising reverting to a previous version of the software module.

9. A system, comprising:

a memory; and

a processing device, operatively coupled to the memory, is configured to:

retrieve, from a repository, a software test and a software module;

execute the software test against the software module a first number of times, observing a first number of failures, to obtain a first flakiness value;

execute the software test against the software module a second number of times, observing a second number of failures, to obtain a second flakiness value;

in response to a convergence criterion between the first flakiness value and the second flakiness value not being met, execute the software test against the software module a third number of times, observe a third flakiness value; and

in response to a difference between one of the first flakiness value, the second flakiness value, or the third flakiness value and a set of historical flakiness values exceeding a threshold, create a defect record.

10. The system of claim 9 , the processing device further to update the set of historical flakiness values with at least one of the first flakiness value, the second flakiness value, or the third flakiness value.

11. The system of claim 9 , the processing device further to execute the software test in response to a change in the software module.

12. The system of claim 11 , the processing device further to back out the change in the software module in response to the difference between at least one of the first flakiness value, the second flakiness value, or the third flakiness value and the set of historical flakiness values exceeding the threshold.

13. The system of claim 9 , wherein the executing the software test further comprises:

executing the software test the first number of times, periodically, at a first frequency interval, to obtain the first flakiness value; and

executing the software test the second number of times, periodically, at a second frequency interval, the second frequency interval less than the first frequency interval, to obtain the second flakiness value.

14. The system of claim 13 , wherein on a condition that the second flakiness value exceeds the first flakiness value by a second threshold, create the defect record.

15. A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:

retrieve, from a repository, a software test and a software module;

execute the software test against the software module a first number of times, observing a first number of failures, to obtain a first flakiness value;

execute the software test against the software module a second number of times, observing a second number of failures, to obtain a second flakiness value;

in response to a convergence criterion between the first flakiness value and the second flakiness value not being met, executing the software test against the software module a third number of times, observe a third flakiness value; and

in response to a difference between one of the first flakiness value, the second flakiness value, or the third flakiness value and a set of historical flakiness values exceeding a threshold, create a defect record.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the threshold is at least 0.001.

17. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions further cause the processing device to update the set of historical flakiness values with at least one of the first flakiness value, the second flakiness value, or the third flakiness value.