Automated testing of program code under development

1. A method comprising:

winnowing a plurality of automated test failures that result from automated testing of program code under development to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;

clustering the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect;

winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failure clusters that each individually correspond to a different automated test code defect are removed; and

outputting the automated test failure clusters that remain after winnowing.

2. The method of claim 1 , wherein outputting the automated test failure clusters comprises:

outputting a ratio of a number of the automated test failure clusters to a number of manually identified program code defects within the program code, as a ratio of a number of automated testing-identified program code defects within the program code to the number of manually identified program code defects.

3. The method of claim 2 , wherein outputting the automated test failure clusters further comprises:

outputting a graph of the ratio of the automated testing-identified program code defects to the number of manually identified program code defects over time.

4. The method of claim 2 , further comprising:

modifying the automated testing of the program code under development to increase the ratio of the ratio of the automated testing-identified program code defects to the number of manually identified program code defects.

5. The method of claim 1 , further comprising:

modifying the automated testing of the program code under development to improve the automated testing.

6. The method of claim 1 , wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:

winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to an environment in which the automated testing of the program code under development was conducted, the issues being failures within hardware of the environment, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

7. The method of claim 1 , wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:

winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to instability of the automated testing of the program code under development, including an inability of the automated test code in handling race conditions, resulting in multiple operations being performed simultaneously instead of in sequence, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

8. The method of claim 1 , wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:

winnowing the automated test failures to include the automated test failures identified as regression automated test failures.

9. The method of claim 1 , wherein winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect comprises:

for each automated test failure cluster, determining whether modification of the automated test code that was executed to perform the automated testing of the program code under development resolved the automated test failures of the automated test failure cluster; and

excluding from the automated failure test clusters each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code, in that each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code is due to an automated test code defect and not due to a program code defect.

10. The method of claim 9 , wherein, for each automated test failure cluster, determining whether modification of the automated test code resolved the automated test failures of the automated test failure cluster comprises:

determining when the automated test failures of the automated test failure cluster last occurred during the automated testing of the program code;

determining when the automated test failures of the automated test failure cluster first no longer occurred during the automated testing of the program code;

determining whether the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred; and

responsive to determining that the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred, determining that modification of the automated test code resolved the automated test failures of the automated test failure cluster.

11. A non-transitory computer-readable data storage medium storing program code executable by a processor to:

winnow a plurality of automated test failures that result from automated testing of program coder under development to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;

cluster the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect;

winnow the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failures that each individually correspond to a different automated test code defect are removed; and

output the automated test failure clusters that remain after.

12. The non-transitory computer-readable data storage medium of claim 11 , wherein the processor is to winnow the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects by:

winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to an environment in which the automated testing of the program code under development was conducted, the issues being failures within hardware of the environment, such that the automated test failures that result from the issues are not indicative of defects within the program code under development; and

winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to instability of the automated testing of the program code under development, including an inability of the automated test code in handling race conditions, resulting in multiple operations being performed simultaneously instead of in sequence, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

13. The non-transitory computer-readable data storage medium of claim 11 , wherein the processor is to winnow the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect by:

for each automated test failure cluster, determining whether modification of automated test code that was executed to perform the automated testing of the program code under development resolved the automated test failures of the automated test failure cluster; and

excluding from the automated test failure clusters each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code, in that each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code is due to an automated test code defect and not due to a program code defect.

14. The non-transitory computer-readable data storage medium of claim 13 , wherein, for each automated test failure cluster, determining whether modification of the automated test code resolved the automated test failures of the automated test failure cluster comprises:

determining when the automated test failures of the automated test failure cluster last occurred during the automated testing of the program code;

determining when the automated test failures of the automated test failure cluster first no longer occurred during the automated testing of the program code;

determining whether the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred; and

responsive to determining that the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred, determining that modification of the automated test code resolved the automated test failures of the automated test failure cluster.

15. The non-transitory computer-readable data storage medium of claim 11 , wherein the processor is to output the automated test failure clusters by:

outputting a ratio of a number of the automated test failure clusters to a number of manually identified program code defects within the program code, as a ratio of a number of automated testing-identified program code defects within the program code to the number of manually identified program code defects.

16. The non-transitory computer-readable data storage medium of claim 15 , wherein the processor is to output the automated test failure clusters by further:

outputting a graph of the ratio of the automated testing-identified program code defects to the number of manually identified program code defects over time.

17. A system comprising:

a processor; and

a memory storing program code executable by the processor, the program code comprising:

automated testing analysis program code to:

winnow a plurality of automated test failures that result from automated testing of program coder under development to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;

cluster the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect;

winnow the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failures that each individually correspond to a different automated test code defect are removed; and

output the automated test failure clusters that remain after winnowing.

18. The system of claim 17 , wherein the program code further comprises:

automated testing program code to perform the automated testing on the program code under development; and

development program code to modify the program code under development.

19. The system of claim 17 , wherein the automated testing analysis program code is to winnow the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects by:

winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to an environment in which the automated testing of the program code under development was conducted, the issues being failures within hardware of the environment, such that the automated test failures that result from the issues are not indicative of defects within the program code under development; and

winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to instability of the automated testing of the program code under development, including an inability of the automated test code in handling race conditions, resulting in multiple operations being performed simultaneously instead of in sequence, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

20. The system of claim 17 , wherein the automated testing analysis program code is to winnow the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect by:

for each automated test failure cluster, determining whether modification of automated test code that was executed to perform the automated testing of the program code under development resolved the automated test failures of the automated test failure cluster; and

excluding from the automated test failure clusters each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code, in that each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code is due to an automated test code defect and not due to a program code defect.