Root cause analysis of non-deterministic tests
Example implementations relate to root cause analysis of non-deterministic tests. In response to detecting a non-deterministic test of an application under test, some implementations may cause rerun executions of the non-deterministic test. Some implementations may also capture data during the rerun executions. The data may include application data, test data, and environment data. Some implementations may also analyze the captured data to determine a root cause of a failure of the non-deterministic test. The analysis may be based on a comparison between failed rerun executions and passed rerun executions.
1. A system for root cause analysis of non-deterministic tests comprising:
a test rerun execution engine to cause, in response to detecting a non-deterministic test of an application under test, rerun executions of the non-deterministic test;
a data capture engine to capture data during the rerun executions, the data including application data, test data, and environment data; and
a root cause analysis engine to analyze the captured data to determine a root cause of a failure of the non-deterministic test by identifying patterns in the captured data, the analysis being based on a comparison between failed rerun executions of the rerun executions and passed rerun executions of the rerun executions,
wherein the patterns are identified based on an analysis of the captured data, the analysis including at least one of:
a comparison between a number of the failed rerun executions that perform a first particular flow and a number of the passed rerun executions that perform the first particular flow;
a comparison between a number of the failed rerun executions that perform a second particular flow and pass a first parameter and a number of the passed rerun executions that perform the second particular flow and pass a second parameter;
a comparison between a size of a first thread count of the failed rerun executions to a size of a second thread count of the passed rerun executions; or
a comparison between an environment related to the failed rerun executions and an environment related to the passed rerun executions.
2. The system of claim 1 , wherein the test rerun execution engine detects the non-deterministic test, the non-deterministic test being detected based on whether some original executions of the non-deterministic test have passed and some original executions of the non-deterministic test have failed without any changes to a code of the application under test or a code of the non-deterministic test.
3. The system of claim 1 , wherein the test rerun execution engine determines a number of the rerun executions to execute based on at least one of a length of time to perform the rerun executions and an amount of resources available to perform the rerun executions.
4. The system of claim 1 , wherein capturing the data includes:
identifying a portion of the application under test into which insertion of instrumentation code is desired, the portion of the application being identified based on a code of the application under test being tested by the non-deterministic test; and
instrumenting the portion of the application under test with the instrumentation code.
5. The system of claim 1 , wherein:
the application data includes data related to the application under test, the data including execution flows of the application under test performed during the rerun executions;
the test data includes data related to the rerun executions of the test, the test data including test status data; and
the environment data includes data related to the environment of the application under test, the environment data including data related to hardware and programming that execute the application under test.
6. The system of claim 1 , wherein identifying the patterns comprises:
comparing attributes of the failed rerun executions to attributes of the passed rerun executions;
determining, based on the comparison, whether a difference between the attributes of the failed rerun executions and the attributes of the passed rerun executions is statistically significant;
if the difference is statistically significant, identifying the difference as a pattern; and
if the difference is not statistically significant, not identifying the difference as a pattern.
7. A computer-implemented method comprising:
receiving, via a test rerun execution engine, a first request from a first portion of code to capture data during rerun executions of a non-deterministic test of an application under test;
in response to receiving the first request, and via a data capture engine:
instrumenting code of the application under test, the instrumented code including instructions to capture the data, the data including metadata related to the application under test and an environment of the application under test; and
executing the instrumented code to capture the data;
receiving, via a root cause analysis engine, a second request from a second portion of code to analyze the captured data; and
in response to receiving the second request, determining, via the root cause analysis engine, a root cause of a failure of the non-deterministic test by analyzing the captured data to identify patterns in the captured data that are common among failed rerun executions of the non-deterministic test and not common among passed rerun executions of the non-deterministic test,
wherein the patterns are identified based on an analysis of the captured data, the analysis including at least one of:
a comparison between a number of the failed rerun executions that perform a first particular flow and a number of the passed rerun executions that perform the first particular flow;
a comparison between a number of the failed rerun executions that perform a second particular flow and pass a first parameter and a number of the passed rerun executions that perform the second particular flow and pass a second parameter;
a comparison between a size of a first thread count of the failed rerun executions to a size of a second thread count of the passed rerun executions; or
a comparison between an environment related to the failed rerun executions and an environment related to the passed rerun executions.
8. The computer-implemented method of claim 7 , wherein identifying the patterns includes:
comparing attributes of the failed rerun executions to attributes of the passed rerun executions; and
determining, based on the comparison, whether a difference between the attributes of the failed rerun executions and the passed rerun executions is statistically significant.
9. The computer-implemented method of claim 7 , wherein:
the captured data is stored in a data repository; and
analyzing the captured data includes accessing the captured data stored in the data repository.
10. The computer-implemented method of claim 7 , wherein analyzing the captured data includes causing the display of information related to the patterns.
11. A non-transitory machine-readable storage medium including instructions which, when executed by a processor, cause the processor to:
store captured data in a data repository, the captured data including:
rerun execution data related to a first set of rerun executions of a non-deterministic test of an application under test, the rerun execution data including execution flows, test status data, and environment data; and
analysis data related to an analysis of the rerun execution data, the analysis data including patterns in the rerun execution data that are common among failed rerun executions and not common among passed rerun executions,
wherein the patterns are based on an analysis of the rerun execution data, the analysis including at least one of:
a comparison between a number of the failed rerun executions that perform a first particular flow and a number of the passed rerun executions that perform the first particular flow;
a comparison between a number of the failed rerun executions that perform a second particular flow and pass a first parameter and a number of the passed rerun executions that perform the second particular flow and pass a second parameter;
a comparison between a size of a first thread count of the failed rerun executions to a size of a second thread count of the passed rerun executions; or
a comparison between an environment related to the failed rerun executions and an environment related to the passed rerun executions; and
update the analysis data based on a second set of rerun executions of the non-deterministic test.
12. The non-transitory machine-readable storage medium of claim 11 , wherein:
the rerun execution data is a first rerun execution data;
data related to the second set of rerun executions is second rerun execution data; and
updating the analysis data includes identifying patterns in the first rerun execution data and the second rerun execution data that are common among the failed rerun executions in the first set of rerun executions and failed rerun executions in the second set of rerun executions and not common among the passed rerun executions in the first set of rerun executions and passed rerun executions in the second set of rerun executions.
13. The non-transitory machine-readable storage medium of claim 11 , wherein the updating is performed in real-time.