System and method for identifying a faulty component in a spectrum ambiguity group
One embodiment provides a system and method for automatically localizing faults in a software system. During operation, the system executes a number of tests associated with the software system to output a spectrum indicating test-coverage patterns of components within the software system; and identifies, based on the spectrum, an ambiguity group associated with at least one failed test. The ambiguity group includes multiple components having a same coverage pattern. The system instruments the multiple components. Instrumenting a respective component includes inserting instructions for monitoring runtime values of at least one variable. The system identifies a subset of tests within which the instrumented multiple components are active, re-executes the identified subset of tests with the multiple components being instrumented, and identifies one component from the multiple components as being likely to cause the at least one failed test based on results of the re-executed subset of tests.
1. A computer-implemented method for automatically localizing faults in a software system, the method comprising:
executing, by a computer, a number of tests associated with the software system to output a spectrum indicating test-coverage patterns of components within the software system;
identifying, based on the spectrum, an ambiguity group associated with at least one failed test, wherein the ambiguity group comprises multiple components having a same coverage pattern;
instrumenting the multiple components, wherein instrumenting a respective component comprises inserting instructions for monitoring runtime values of at least one variable;
identifying a subset of tests within which the instrumented multiple components are active;
re-executing the identified subset of tests with the multiple components being instrumented; and
identifying one component from the multiple components as being likely to cause the at least one failed test based on results of the re-executed subset of tests.
2. The computer-implemented method of claim 1 , further comprising:
identifying one or more states associated with a component based on the monitored runtime values of the at least one variable; and
determining coverage patterns of the component in different states.
3. The computer-implemented method of claim 2 , wherein identifying the one or more states comprises partitioning the monitored runtime values of the at least one variable into a number of subsets, each subset corresponding to a state.
4. The computer-implemented method of claim 3 , wherein partitioning the monitored runtime values of the at least one variable comprises one or more of:
applying a landmark-based partitioning technique;
applying a cluster-based partitioning technique; and
applying a label-based partitioning technique.
5. The computer-implemented method of claim 2 , further comprising:
generating an augmented spectrum based on results of the re-executed subset of tests, wherein a respective row in the spectrum corresponds to a component in an identified state, and wherein a respective column in the spectrum corresponds to a test.
6. The computer-implemented method of claim 2 , wherein the at least one variable comprises a numerical variable, and wherein a respective state corresponds to a predetermined numerical range.
7. The computer-implemented method of claim 2 , wherein the at least one variable comprises a pointer variable, and wherein a respective state corresponds to the pointer variable being null or not null.
8. The computer-implemented method of claim 1 , wherein executing the tests comprises:
performing a number of iterations; and
in each iteration,
decreasing instrumentation granularity;
filtering out components that are less likely to cause the at least one failed test based on results of a previous iteration; and
executing tests that are affected by remaining components.
9. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for automatically localizing faults in a software system, the method comprising:
executing a number of tests associated with the software system to output a spectrum indicating test-coverage patterns of components within the software system;
identifying, based on the spectrum, an ambiguity group associated with at least one failed test, wherein the ambiguity group comprises multiple components having a same coverage pattern;
instrumenting the multiple components, wherein instrumenting a respective component comprises inserting instructions for monitoring runtime values of at least one variable;
identifying a subset of tests within which the instrumented multiple components are active;
re-executing the identified subset of tests with the multiple components being instrumented; and
identifying one component from the multiple components as being likely to cause the at least one failed test based on results of the re-executed subset of tests.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the method further comprises:
identifying one or more states associated with a component based on the monitored runtime values of the at least one variable; and
determining coverage patterns of the component in different states.
11. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the one or more states comprises partitioning the monitored runtime values of the at least one variable into a number of subsets, each subset corresponding to a state.
12. The non-transitory computer-readable storage medium of claim 11 , wherein partitioning the monitored runtime values of the at least one variable comprises one or more of:
applying a landmark-based partitioning technique;
applying a cluster-based partitioning technique; and
applying a label-based partitioning technique.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the method further comprises:
generating an augmented spectrum based on results of the re-executed subset of tests, wherein a respective row in the spectrum corresponds to a component in an identified state, and wherein a respective column in the spectrum corresponds to a test.
14. The non-transitory computer-readable storage medium of claim 10 , wherein the at least one variable comprises a numerical variable, and wherein a respective state corresponds to a predetermined numerical range.
15. The non-transitory computer-readable storage medium of claim 10 , wherein the at least one variable comprises a pointer variable, and wherein a respective state corresponds to the pointer variable being null or not null.
16. The non-transitory computer-readable storage medium of claim 9 , wherein executing the tests comprises:
performing a number of iterations; and
in each iteration,
decreasing instrumentation granularity;
filtering out components that are less likely to cause the at least one failed test based on results of a previous iteration; and
executing tests that are affected by remaining components.
17. A computer system for automatically localizing faults in a software system, the computer system comprising:
a processor; and
a non-transitory storage device storing instructions that when executed by the processor cause the processor to perform a method, the method comprising:
executing a number of tests associated with the software system to output a spectrum indicating test-coverage patterns of components within the software system;
identifying, based on the spectrum, an ambiguity group associated with at least one failed test, wherein the ambiguity group comprises multiple components having a same coverage pattern;
instrumenting the multiple components, wherein instrumenting a respective component comprises inserting instructions for monitoring runtime values of at least one variable;
identifying a subset of tests within which the instrumented multiple components are active;
re-executing the identified subset of tests with the multiple components being instrumented; and
identifying one component from the multiple components as being likely to cause the at least one failed test based on results of the re-executed subset of tests.
18. The computer system of claim 17 , wherein the method further comprises:
identifying one or more states associated with a component based on the monitored runtime values of the at least one variable; and
determining coverage patterns of the component in different states.
19. The computer system of claim 18 , wherein identifying the one or more states comprises partitioning the monitored runtime values of the at least one variable into a number of subsets, each subset corresponding to a state.
20. The computer system of claim 19 , wherein partitioning the monitored runtime values of the at least one variable comprises one or more of:
applying a landmark-based partitioning technique;
applying a cluster-based partitioning technique; and
applying a label-based partitioning technique.