Systems and methods for monitoring progression of software versions and detection of anomalies
Disclosed herein are system, method, and computer program product embodiments for detecting anomalies during software testing. The methods include generating a plurality of test reports for the software program by executing one or more test cases on a plurality of versions of the software program, generating a control chart based on the plurality of test reports, generating an alert when at least one testing characteristic includes an anomaly over the plurality of versions of the software program as determined based on the control chart. The control chart includes a plot associated with at least one testing characteristic of the software program, and a historical context associated with execution of the one or more test cases on the plurality of versions of the software program.
1. A method for testing of a software program to be executed on an autonomous vehicle, the method comprising, by a processor:
performing autonomous vehicle simulation tests by executing one or more test cases on a plurality of versions of the software program, each test case comprising a simulation scenario configuration;
monitoring at least one testing characteristic of the software program during execution of the one or more test cases, the at least one testing characteristic comprising a feature of the one or more test cases which changes over time in view of changes to the software program;
generating, based on the monitoring, a plurality of test reports containing metrics corresponding to results of test case executions for the software program;
generating (i) a first control chart based on ones of the plurality of test reports that are associated with a control group of test cases and (ii) a second control chart based on ones of the plurality of reports that are associated with an impacted group of test cases, wherein each of the first and second control charts comprises:
a plot of the monitored at least one testing characteristic of the software program over time, and
a historical context associated with execution of the one or more test cases on the plurality of versions of the software program;
identifying an anomalous behavior of the at least one testing characteristic based on an unusual pattern in at least the second control chart;
determining a cause of the anomalous behavior by comparing the second control chart to the first control chart;
generating an alert that the at least one testing characteristic has the anomalous behavior over the plurality of versions of the software program; and
modifying control system operations and/or sensor design of the autonomous vehicle based on the alert;
wherein the control group of test cases includes one or more first test cases that are executed on portions of the software program that remain unchanged over the plurality of versions of the software program, and the impacted group of test cases includes one or more second test cases that are executed on portions of the software program that change over at least some of the plurality of versions of the software program.
2. The method of claim 1 , wherein generating the control chart comprises generating a univariate control chart, the univariate control chart comprising:
the plot associated with a testing characteristic of the software program,
a control line,
an upper control limit, and
a lower control limit.
3. The method of claim 2 , wherein the control line, the upper control limit, and the lower control limit are generated based on the historical context associated with execution of the one or more test cases on the plurality of versions of the software program.
4. The method of claim 2 , further comprising determining that the testing characteristic includes an anomaly in response to one or more values of the testing characteristic lying outside upper control limit or the lower control limit.
5. The method of claim 1 , wherein generating the control chart comprises generating a multivariate control chart, the multivariate control chart comprising the plot associated with a first testing characteristic and a second testing characteristic.
6. The method of claim 5 , further comprising determining that the first testing characteristic and the second testing characteristic include an anomaly in response to a multivariate distance from a control mean being greater than a threshold.
7. The method of claim 1 , further comprising determining a degree of the anomalous behavior.
8. The method of claim 1 , wherein the at least one testing characteristic comprises at least one of the following: a pass percentage, a failure percentage, a pass/fail ratio, a system boundary over a simulation scenario, error modes, numerical performance metrics, or system latencies.
9. The method of claim 1 , further comprising:
identifying a group of test cases based on a functionality of the software program being tested, submodules/modalities of the software programs being tested, base scenarios being used for simulation tests, or failure modes being tested;
wherein the plot of the control chart includes a plot of the monitored at least one testing characteristic over the plurality of versions of the software program for the identified group of test cases.
10. A system for testing of a software program to be executed on an autonomous vehicle, the system comprising:
at least one processor; and
programming instructions stored in a memory and configured to cause the processor to:
perform autonomous vehicle simulation tests by executing one or more test cases on a plurality of versions of the software program, each test case comprising a simulation scenario configuration;
monitor at least one testing characteristic of the software program during execution of the one or more test cases, the at least one testing characteristic comprising a feature of the one or more test cases which changes over time in view of changes to the software program;
generate a plurality of test reports containing metrics corresponding to text case executions for the software program,
generate (i) a first control chart based on ones of the plurality of test reports that are associated with a control group of test cases and (ii) a second control chart based on ones of the plurality of reports that are associated with an impacted group of test cases, wherein each of the first and second control charts comprises:
a plot of the monitored at least one testing characteristic of the software program over time, and
a historical context associated with execution of the one or more test cases on the plurality of versions of the software program, and
identify an anomalous behavior of the at least one testing characteristic based on an unusual pattern in at least the second control chart;
determine a cause of the anomalous behavior by comparing the second control chart to the first control chart;
generate an alert, based on the control chart, that the at least one testing characteristic has the anomalous behavior over the plurality of versions of the software program; and
modify control system operations and/or sensor design of the autonomous vehicle based on the alert;
wherein the control group of test cases includes one or more first test cases that are executed on portions of the software program that remain unchanged over the plurality of versions of the software program, and the impacted group of test cases includes one or more second test cases that are executed on portions of the software program that change over at least some of the plurality of versions of the software program.
11. The system of claim 10 , wherein the programming instructions that are configured to cause the processor to generate the control chart comprise programming instructions to cause the processor to generate a univariate control chart, the univariate control chart comprising:
the plot associated with a testing characteristic of the software program,
a control line,
an upper control limit, and
a lower control limit.
12. The system of claim 11 , wherein the control line, the upper control limit, and the lower control limit are generated based on the historical context associated with execution of the one or more test cases on the plurality of versions of the software program.
13. The system of claim 11 , further comprising programming instructions that are configured to cause the processor to determine that the testing characteristic includes an anomaly in response to one or more values of the testing characteristic lying outside upper control limit or the lower control limit.
14. The system of claim 10 , wherein the programming instructions that are configured to cause the processor to generate the control chart comprise programming instructions to cause the processor to generate a multivariate control chart, the multivariate control chart comprising the plot associated with a first testing characteristic and a second testing characteristic.
15. The system of claim 10 , further comprising programming instructions that are configured to cause the processor to determine a degree of the anomalous behavior.
16. The system of claim 10 , wherein the at least one testing characteristic comprises at least one of the following: a pass percentage, a failure percentage, a pass/fail ratio, a system boundary over a simulation scenario, error modes, numerical performance metrics, or system latencies.
17. The system of claim 10 , wherein:
the processor is further caused to identify a group of test cases based on a functionality of the software program being tested, submodules/modalities of the software programs being tested, base scenarios being used for simulation tests, or failure modes being tested; and
the plot of the control chart includes a plot of the monitored at least one testing characteristic over the plurality of versions of the software program for the identified group of test cases.
18. A computer program product comprising a non-transitory computer-readable medium that stores instructions that, when executed by a computing device, will cause the computing device to perform operations comprising:
performing autonomous vehicle simulation tests by executing one or more test cases on a plurality of versions of the software program, each test case comprising a simulation scenario configuration;
monitoring at least one testing characteristic of the software program during execution of the one or more test cases, the at least one testing characteristic comprising a feature of the one or more test cases which changes over time in view of changes to the software program;
generating, based on the monitoring, a plurality of test reports for the software program;
generating (i) a first control chart based on ones of the plurality of test reports that are associated with a control group of test cases and (ii) a second control chart based on the plurality of reports that are associated with an impacted group of test cases, wherein each of the first and second control charts comprises:
a plot of the monitored at least one testing characteristic of the software program over time, and
a historical context associated with execution of the one or more test cases on the plurality of versions of the software program; and
identifying an anomalous behavior of the at least one testing characteristic based on an unusual pattern in at least the second control chart;
determine a cause of the anomalous behavior by comparing the second control chart to the first control chart;
generating an alert that the at least one testing characteristic has the anomalous behavior over the plurality of versions of the software program; and
modifying control system operations and/or sensor design of the autonomous vehicle based on the alert;
wherein the control group of test cases includes one or more first test cases that are executed on portions of the software program that remain unchanged over the plurality of versions of the software program, and the impacted group of test cases includes one or more second test cases that are executed on portions of the software program that change over at least some of the plurality of versions of the software program.