Statistical analysis of sampled profile data in the identification of significant software test performance regressions
Sampled profile data provides information about processor activity during a test. Processor activity can be analyzed to determine an amount of processor resources used to execute the various functions, modules, and processes associated with a tested software activity. Statistical methods can be applied to the resource data from multiple test runs to determine whether a significant regression has occurred between a baseline test pass and a daily test pass. By collecting data at the function, module and process levels, significant regressions may be uncovered at any of the levels. Regressions may also be ranked according to their importance, which allows for identification and notification of development teams responsible for significant regressions.
1. In a system for analyzing computer test results, said test results comprising a plurality of sampled processor events, a method comprising:
computing an amount of processor resources used to execute each memory address in each function, a memory address comprising a subset of a function;
computing an amount of processor resources used to execute each function in each module based on the amount of processor resources used to execute each memory address of each function, a function comprising a subset of a module;
computing an amount of processor resources used to execute each module in said process based on the amount of processor resources used to execute each function of each module, a module comprising a subset of a process;
computing an amount of processor resources used to execute the process based on the amount of processor resources used to execute each module of the process;
for each amount of processor resources used to execute each process computed, each amount of processor resources used to execute each module computed, each amount of processor resources used to execute each function computed;
computing an error amount associated with said amount of processor resources;
computing said amount of processor resources and said error amount with amounts from a previous test; and
finding a significant regression if said amount of processor resources differs from an amount of processor resources from the previous test, after accounting for said error amount and an error amount from the previous test.
2. The method of claim 1 , further comprising ranking the regression among a plurality of regressions according to an amount of difference between said amount of processor resources and the amount of processor resources from the previous test.
3. The method of claim 1 , wherein computing an amount of processor resources comprises computing a mean amount of processor resources used to execute the function, and where the mean is computed using results from multiple runs of a test.
4. The method of claim 3 , wherein said computing an error amount comprises computing a statistical error amount based at least in part on a number of runs of the test.
5. The method of claim 1 , further comprising computing an amount of processor resources that was used to execute a module by combining one or more amounts of processor resources that were used to execute one or more functions associated with the module.
6. The method of claim 5 , further comprising computing at least one module error amount.
7. The method of claim 6 , further comprising finding a module regression if said amount of processor resources that was used to execute a module differs from an amount of processor resources that was used to execute the module from the previous test, after accounting for said error amount and an error amount from the previous test.
8. A system for testing computer software, comprising:
a processor;
a test harness for causing a plurality of test runs, wherein a test run comprises executing tested software on the processor;
a profiler for sampling processor activity during executing of said tested software, thereby generating sample data;
software for calculating from said sample data an amount of processor resources in each test run used to execute each memory address in each function, a memory address comprising a subset of a function;
software for calculating from said sample data an amount of processor resources in each test run used to execute each function in each module based on the amount of processor resources used to execute each memory address of each function, a function comprising a subset of a module;
software for calculating from said sample data an amount of processor resources in each test run used to execute each module in said process based on the amount of processor resources used to execute each function of each module, a module comprising a subset of a process;
software for calculating from said sample data an amount of processor resources in each test run used to execute the process based on the amount of processor resources used to execute each module of the process;
software for calculating from said each amount of processor resources in each test run a mean amount of processor resources dedicated to executing the corresponding function, module or process and an error amount associated with said mean amount;
software for comparing said mean amount and said error amount to a previous mean amount and a previous error amount from a previous test to determine if there is a significant regression in the amount of processor resources dedicated to executing the function module or process.
9. The system of claim 8 , wherein said software for comparing comprises instructions for ranking regressions according to an amount of difference between said mean amount and said previous mean amount, while accounting for said error amount and said and previous error amount.
10. The system of claim 8 , wherein said software for calculating from said sample data an amount of processor resources also calculates an amount of processor resources in each test run that was dedicated to executing at least one module associated with the tested software.
11. The system of claim 8 , wherein said error amount represents a 75%-99% confidence in said mean amount of processor resources dedicated to executing the at least one function.
12. The system of claim 8 , further comprising software for discovering from said sample data at least on function, module, and process associated with the tested software.
13. The system of claim 8 , wherein the profiler is an Event Tracing for WINDOWS® (ETW) profiler.
14. A method to be carried out by a computer for ranking software test regressions, comprising:
calculating a mean amount of processor resources that are dedicated to executing a function, wherein said processor resources are measured in Central Processing Unit (CPU) clock cycles;
calculating an error associated with said mean amount;
calculating a difference between said mean amount and a previous mean amount, wherein said error amount to and a previous error amount are accounted for;
ranking a regression based on said difference; and
filtering the regression if the error amount and the previous error amount are greater than a simple difference between said mean amount and a previous mean amount.
15. The method of claim 14 wherein said calculating a mean amount, calculating an error, calculating a difference, and ranking are also carried out for a module.
16. The method of claim 14 wherein said calculating a mean amount, calculating an error, calculating a difference, and ranking are also carried out for a process.
17. The method of claim 14 wherein said difference is calculated using the equation |D|*(1−E/|D|).
18. The method of claim 14 wherein there is statistically a 75%-99% confidence that a subsequent test would dedicate to executing said function an amount of processor resources determinable from said mean amount and said error amount.
19. The method of claim 14 wherein said processor resources are measured in Central Processing Unit (CPU) clock cycles.