IP Library › Granted Patent US 11,010,285

Granted Patent B2

US 11,010,285 · App. 16/256,388 · Granted May 18, 2021

Fault detection and localization to generate failing test cases using combinatorial test design techniques

Inventors: Andrew Hicks (Wappingers Falls, NY); Dale E. Blue (Poughkeepsie, NY); Ryan Rawlins (New Paltz, NY); Rachel Brill (Haifa, IL)

Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION

G06F11/3684 G06F11/263 G06F11/3676 G06F11/3692

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,010,285
App. No.: 16/256,388
Granted: May 18, 2021
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Systems, methods, and computer-readable media are described for performing fault detection and localization using Combinatorial Test Design (CTD) techniques and generating a regression bucket of test cases that expose a detected fault in a System Under Test (SUT). The SUT may be a hardware system or a software system. Further, the fault detection and localization may be performed while adhering to architectural restrictions on the SUT.

Claims (66)

1. A method for detecting and localizing a fault in a System Under Test (SUT), the method comprising:

modeling inputs to the SUT as a collection of attribute-value pairs;

generating an initial set of test vectors that provides complete n-wise coverage of a test space associated with the collection of attribute-value pairs;

generating an initial set of test cases from the initial set of test vectors;

executing the initial set of test cases to obtain a first set of execution results;

determining, based at least in part on the first set of execution results, that one or more test cases failed execution;

selecting, from the one or more test cases that failed execution, a failing test case to identify a cause of failure, the failing test case tests a first attribute with a first value;

generating a set of new test cases from the selected failing test case, wherein generating the set of new test cases comprises changing the first value of the first attribute in the selected failing test case to a respective value in each new test case that is generated;

executing the set of new test cases to obtain a second set of execution results; and

detecting and localizing the fault based at least in part on the second set of execution results.

2. The method of claim 1 , wherein changing the respective value in the selected failing test case to generate each new test case comprises changing the respective value to an attribute value that is not present in any of the one or more test cases that failed execution.

3. The method of claim 1 , wherein detecting and localizing the fault based at least in part on the second set of execution results comprises:

determining that a particular new test case failed execution; and

determining that the attribute having the first value in the selected failing test case that was changed to obtain the particular new test case does not contribute to the fault.

4. The method of claim 1 , wherein detecting and localizing the fault based at least in part on the second set of execution results comprises:

determining that a first new test case and a second new test case each successfully executed;

determining a second attribute having a second original attribute value in the selected failing test case that was changed to generate the second new test case; and

determining that the first value and the second original attribute value contribute to the fault.

5. The method of claim 4 , wherein the initial set of test vectors provides complete pairwise coverage of the test space, wherein the fault is a pairwise error in the SUT, and wherein determining that the first original attribute value and the second original attribute value contribute to the fault comprises determining that the first original attribute value and the second original attribute value cause the pairwise error.

6. The method of claim 1 , wherein detecting and localizing the fault based at least in part on the second set of execution results comprises determining a particular combination of attribute values that causes the fault.

7. The method of claim 6 , further comprising:

generating a regression bucket of failing test cases based at least in part on the particular combination of attribute values that causes the fault, wherein the regression bucket covers all possible combinations of the collection of attribute-value pairs that include the particular combination of attribute values that causes the fault.

8. A system for detecting and localizing a fault in a System Under Test (SUT), the system comprising:

at least one processor; and

at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:

model inputs to the SUT as a collection of attribute-value pairs;

generate an initial set of test vectors that provides complete n-wise coverage of a test space associated with the collection of attribute-value pairs;

generate an initial set of test cases from the initial set of test vectors;

execute the initial set of test cases to obtain a first set of execution results;

determine, based at least in part on the first set of execution results, that one or more test cases failed execution;

select, from the one or more test cases that failed execution, a failing test case to identify a cause of failure, the failing test case tests a first attribute with a first value;

generate a set of new test cases from the selected failing test case, wherein generating the set of new test cases comprises changing the first value of the first attribute in the selected failing test case to a respective value in each new test case that is generated;

execute the set of new test cases to obtain a second set of execution results; and

detect and localize the fault based at least in part on the second set of execution results.

9. The system of claim 8 , wherein the at least one processor is configured to change the respective value in the selected failing test case to generate each new test case by executing the computer-executable instructions to change the respective value to an attribute value that is not present in any of the one or more test cases that failed execution.

10. The system of claim 8 , wherein the at least one processor is configured to detect and localize the fault based at least in part on the second set of execution results by executing the computer-executable instructions to:

determine that a particular new test case failed execution; and

determine that the attribute having the first value in the selected failing test case that was changed to obtain the particular new test case does not contribute to the fault.

11. The system of claim 8 , wherein the at least one processor is configured to detect and localize the fault based at least in part on the second set of execution results by executing the computer-executable instructions to:

determine that a first new test case and a second new test case each successfully executed;

determine a second attribute having a second original attribute value in the selected failing test case that was changed to generate the second new test case; and

determine that the first value and the second original attribute value contribute to the fault.

12. The system of claim 11 , wherein the initial set of test vectors provides complete pairwise coverage of the test space, wherein the fault is a pairwise error in the SUT, and wherein the at least one processor is configured to determine that the first original attribute value and the second original attribute value contribute to the fault by executing the computer-executable instructions to determine that the first original attribute value and the second original attribute value cause the pairwise error.

13. The system of claim 11 , wherein the at least one processor is configured to detect and localize the fault based at least in part on the second set of execution results by executing the computer-executable instructions to determine a particular combination of attribute values that causes the fault.

14. The system of claim 13 , wherein the at least one processor is further configured to execute computer-executable instructions to:

generate a regression bucket of failing test cases based at least in part on the particular combination of attribute values that causes the fault, wherein the regression bucket covers all possible combinations of the collection of attribute-value pairs that include the particular combination of attribute values that causes the fault.

15. A computer program product for detecting and localizing a fault in a System Under Test (SUT), the computer program product comprising a storage medium readable by a processing circuit, the storage medium storing instructions executable by the processing circuit to cause a method to be performed, the method comprising: