Identifying root causes of test failures
A disclosed method defines root cause failure categories for a test case and associates each category with a corresponding test configuration property. Test configuration property information, indicative of the test configuration properties, are embedded in test case metadata. After generating test results, including test script messages, the test script messages assessed to identify the “N” most significant keywords in the message. The significance of a term in the test script message may be calculated based on an inverse document frequent parameter, independent of the term frequency within the document. Test result groups may then be determined by invoking a suitable clustering algorithm, e.g., a k-means clustering algorithm, to cluster the test script messages based on their corresponding keyword sets. Hypothesis test statistics may then be calculated for each test result group. A most probable root cause may then be identified for some or all of the test result groups.
1 . A computer-implemented method executed by one or more processors coupled to a memory, the method comprising:
responsive to completion of a firmware update of an information handling system, initiating automated regression testing by running a plurality of test cases across a plurality of test configurations and detecting one or more test case failures corresponding to a particular test case, the test cases being instrumented to generate test result metadata indicative of configuration properties for each run;
defining, in the memory, a set of root cause failure categories for the particular test case and storing a table that associates each root cause failure category with a corresponding test configuration property, the table comprising cause-configuration pairs that align root cause categories with respective configuration properties;
embedding, at test execution time, configuration property information in test case metadata such that each test result includes values for the associated configuration properties;
accessing, from the memory, test script error messages
included in the test results and, for each failed run, generating a keyword set that includes N most significant keywords for the corresponding test script error message, the significance of each keyword being computed based solely on an inverse document frequency parameter that is independent of term frequency within the message;
forming, for each failed run, an N-dimensional vector representation of its keyword set and clustering the failed runs by applying a k-means clustering algorithm to the vector representations to produce two or more test result groups, and for each test result group including corresponding passed runs to enable statistical dependency evaluation;
for each test result group, constructing, in the memory, a contingency table between configuration property values from the embedded metadata and pass/fail counts for runs in the group, and computing a chi-squared statistic and corresponding p-value for each configuration property;
automatically identifying, by the one or more processors, a most probable root cause for the particular test case by selecting, for at least one test result group, the root cause failure category associated with the configuration property whose chi-squared test exhibits statistical significance at a confidence level of at least 95%; and
storing, in the memory, an electronic record that maps the test result group to the identified most probable root cause together with the corresponding configuration property and p-value, for use in subsequent troubleshooting of the information handling system.
2 . The method of claim 1 , wherein each test script error message to a corresponding one of the test case failures.
3 . The method of claim 1 , wherein the root cause categories include any one or more of: firmware issue, testbed issue, and test script issue.
4 . The method of claim 1 wherein the test configuration properties include any one or more of: a firmware version, indicative of a version of firmware stored in a device under test (DUT), a testbed identifier (ID), and a test case script version.
5 . The method of claim 1 , wherein the test script error include at least one of: one or more assertion messages and one or more annotation messages.
6 . The method of claim 1 , wherein N is an integer greater than or equal to 3 and less than 5.
7 . The method of claim 1 , wherein calculating hypothesis test statistics for two or more test result groups comprises calculating a chi square statistic for each test result group and each configuration property.
8 . An information handling system, comprising:
a central processing unit (CPU); and
a non-transitory computer readable medium including processor-executable instructions that, when executed by the CPU, cause the information handling system to perform operations comprising:
responsive to completion of a firmware update of an information handling system, initiating automated regression testing by running a plurality of test cases across a plurality of test configurations and detecting one or more test case failures corresponding to a particular test case, the test cases being instrumented to generate test result metadata indicative of configuration properties for each run;
defining, in the memory, a set of root cause failure categories for the particular test case and storing a table that associates each root cause failure category with a corresponding test configuration property, the table comprising cause-configuration pairs that align root cause categories with respective configuration properties;
embedding, at test execution time, configuration property information in test case metadata such that each test result includes values for the associated configuration properties;
accessing, from the memory, test script error messages included in the test results and, for each failed run, generating a keyword set that includes N most significant keywords for the corresponding test script error message, the significance of each keyword being computed based solely on an inverse document frequency parameter that is independent of term frequency within the message;
forming, for each failed run, an N-dimensional vector representation of its keyword set and clustering the failed runs by applying a k-means clustering algorithm to the vector representations to produce two or more test result groups, and for each test result group including corresponding passed runs to enable statistical dependency evaluation;
for each test result group, constructing, in the memory, a contingency table between configuration property values from the embedded metadata and pass/fail counts for runs in the group, and computing a chi-squared statistic and corresponding p-value for each configuration property;
automatically identifying, by the one or more processors, a most probable root cause for the particular test case by selecting, for at least one test result group, the root cause failure category associated with the configuration property whose chi-squared test exhibits statistical significance at a confidence level of at least 95%; and
storing, in the memory, an electronic record that maps the test result group to the identified most probable root cause together with the corresponding configuration property and p-value, for use in subsequent troubleshooting of the information handling system.
9 . The information handling system of claim 8 , wherein each test script error message to a corresponding one of the test case failures.
10 . The information handling system of claim 8 , wherein the root cause categories include any one or more of: firmware issue, testbed issue, and test script issue.
11 . The information handling system of claim 10 , wherein the test configuration properties include any one or more of: a firmware version, indicative of a version of firmware stored in a device under test (DUT), a testbed identifier (ID), and a test case script version.
12 . The information handling system of claim 8 , wherein the test script error messages include at least one of: one or more assertion messages and one or more annotation messages.
13 . The information handling system of claim 8 , wherein identifying a keyword set for a test script error message comprises identifying the N most significant words in the test script error message, wherein N is an integer greater than or equal to 3 and less than 5.
14 . The information handling system of claim 8 , wherein calculating hypothesis test statistics for two or more test result groups comprises calculating a chi square statistic for each test result group and each configuration property.